Acoustic Coordination of Audio Sources

ABSTRACT

An audio/video (A/V) hub that coordinates playback of audio content is described. In particular, the A/V hub may calculate current time offsets between clocks in electronic devices (such as electronic devices that include speakers) and a clock in the A/V hub based on measured sound corresponding to one or more acoustic-characterization patterns, one or more times when the electronic devices output the sound and the one or more acoustic-characterization patterns. Then, the A/V hub may transmit, to the electronic devices, one or more frames that include audio content and playback timing information, which may specify playback times when the electronic devices are to playback the audio content based on the current time offsets. Moreover, the playback times of the electronic devices may have a temporal relationship so that the playback of the audio content by the electronic devices is coordinated.

CROSS REFERENCE TO RELATED APPLICATIONS

This application claims priority under 35 U.S.C. 119(e) to U.S.Provisional Application Ser. No. 62/433,237, “Wireless Coordination ofAudio Sources,” by Gaylord Yu, filed on Dec. 13, 2016, the contents ofwhich are herein incorporated by reference.

BACKGROUND Field

The described embodiments relate to a communication technique. Morespecifically, the described embodiments include a communicationtechnique that acoustically coordinates playback times of electronicdevices that output sound.

Related Art

Music often has a significant impact on an individual's emotions andperceptions. This is thought to be a result of connections orrelationships between the areas of the brain that decipher, learn, andremember music with those that produce emotional responses, such as thefrontal lobes and limbic system. Indeed, emotions are thought to beinvolved in the process of interpreting music, and concurrently are veryimportant in the effect of music on the brain. Given this ability ofmusic to ‘move’ a listener, audio quality is often an important factorin user satisfaction when listening to audio content and, moregenerally, when viewing and listening to audio/video (A/V) content.

However, it is often challenging to achieve high audio quality in anenvironment. For example, the acoustic sources (such as loudspeakers)may not be properly placed in the environment. Alternatively oradditionally, a listener may not be located at an ideal position in theenvironment. In particular, in a stereo playback system, the so-called‘sweet spot,’ where the amplitude differences and arrival timedifferences are small enough that an apparent image and localization ofan original sound source are both maintained, is usually limited to afairly small area between the loudspeakers. When the listener is outsidethat area, the apparent image collapses and only one or the otherindependent audio channel output by the loudspeakers may be heard.Furthermore, achieving high audio quality in the environment typicallyplaces strong constraints on synchronization of the loudspeakers.

Consequently, when one or more of these factors is sub-optimal, theacoustic quality in the environment may be degraded. In turn, this mayadversely impact listener satisfaction and the overall user experiencewhen listening to audio content and/or A/V content.

SUMMARY

A first group of described embodiments includes an audio/video (A/V)hub. This A/V hub includes: one or more antennas; and an interfacecircuit that, during operation, communicates with electronic devicesusing wireless communication. During operation, the A/V hub receives,via the wireless communication, frames from the electronic devices,where a given frame includes a transmit time when a given electronicdevice transmitted the given frame. Then, the A/V hub stores receivetimes when the frames were received, where the receive times are basedon a clock in the A/V hub. Moreover, the A/V hub calculates current timeoffsets between clocks in the electronic devices and the clock in theA/V hub based on the receive times and transmit times of the frames.Next, the A/V hub transmits one or more frames that include audiocontent and playback timing information to the electronic devices, wherethe playback timing information specifies playback times when theelectronic devices are to playback the audio content based on thecurrent time offsets. Furthermore, the playback times of the electronicdevices have a temporal relationship so that the playback of the audiocontent by the electronic devices is coordinated.

Note that the temporal relationship may have a non-zero value, so thatat least some of the electronic devices are instructed to playback theaudio content with a phase relative to each other by using differentvalues of the playback times. For example, the different playback timesmay be based on acoustic characterization of an environment thatincludes the electronic devices and the A/V hub. Alternatively oradditionally, the different playback times may be based on a desiredacoustic characteristic in the environment.

In some embodiments, the electronic devices are located at vectordistances from the A/V hub, and the interface circuit determinesmagnitudes of the vector distances based on the transmit times and thereceive times using wireless ranging. Moreover, the interface circuitmay determine angles of the vector distances based on the angle ofarrival of wireless signals associated with the frames that are receivedby the one or more antennas during the wireless communication.Furthermore, the different playback times may be based on the determinedvector distances.

Alternatively or additionally, the different playback times are based onan estimated location of a listener relative to the electronic devices.For example, the interface circuit may: communicate with anotherelectronic device; and calculate the estimated location of the listenerbased on the communication with the other electronic device. Moreover,the A/V hub may include an acoustic transducer that performs soundmeasurements of the environment that includes the A/V hub, and the A/Vhub may calculate the estimated location of the listener based on thesound measurements. Furthermore, the interface circuit may communicatewith other electronic devices in the environment and may receiveadditional sound measurements of the environment from the otherelectronic devices. In these embodiments, the A/V hub calculates theestimated location of the listener based on the additional soundmeasurements. In some embodiments, the interface circuit: performstime-of-flight measurements; and calculates the estimated location ofthe listener based on the time-of-flight measurements.

Note that the electronic devices may be located at non-zero distancesfrom the A/V hub, and the current time offsets may be calculated basedon the transmit times and the receive times using wireless ranging byignoring the distances.

Moreover, the current time offsets may be based on models of clock driftin the electronic devices.

Another embodiment provides a computer-readable storage medium for usewith the A/V hub. This computer-readable storage medium includes aprogram module that, when executed by the A/V hub, cause the A/V hub toperform at least some of the aforementioned operations.

Another embodiment provides a method for coordinating playback of audiocontent. This method includes at least some of the operations performedby the A/V hub.

Another embodiment provides one or more of the electronic devices.

A second group of described embodiments includes an audio/video (A/V)hub. This A/V hub includes: memory that, during operation, storescharacterization information of an environment that includes the A/Vhub; one or more antennas; and an interface circuit that, duringoperation, communicates with an electronic device using wirelesscommunication. During operation, the A/V hub detects, using the wirelesscommunication, the electronic device in the environment. Then, the A/Vhub determines a change condition, where the change condition includes:that the electronic device was not previously detected in theenvironment; and/or a change in a location of the electronic device.When the change condition is determined, the A/V hub transitions into acharacterization mode. During the characterization mode, the A/V hub:provides instructions to the electronic device to playback audio contentat a specified playback time; determines one or more acousticcharacteristics of the environment based on acoustic measurements in theenvironment; and stores the characterization information in the memory,where the characterization information includes the one or more acousticcharacteristics.

Moreover, the characterization information may include: an identifier ofthe electronic device; and the location of the electronic device. Forexample, the location may include a distance between the A/V hub and theelectronic device, and an angle of arrival of wireless signals duringthe wireless communication. Consequently, the change in the location mayinclude a change in: the distance, the angle of arrival, or both. Insome embodiments, the distance is determined using wireless ranging.

Note that the one or more acoustic characteristics may includeinformation specifying: an acoustic transfer function in at least afirst band of frequencies, acoustic loss, acoustic delay, acoustic noisein the environment, ambient sound in the environment, a reverberationtime of the environment, and/or a spectral response in at least a secondband of frequencies.

Furthermore, the A/V hub may calculate the location of the electronicdevice in the environment based on the wireless communication.

Additionally, the interface circuit may communicate with otherelectronic devices in the environment using the wireless communication,and the acoustic measurements may be received from the other electronicdevices. In these embodiments, the one or more acoustic characteristicsmay be determined based on locations of the other electronic devices inthe environment. Note that the A/V hub may: receive the locations of theother electronic devices from the other electronic devices; accesspredetermined locations of the other electronic devices stored in thememory; and determine the locations of the other electronic devicesbased on the wireless communication.

In some embodiments, the A/V hub includes one or more acoustictransducers, and the A/V hub performs the acoustic measurements usingthe one or more acoustic transducers.

Moreover, the A/V hub may: receive a user input; and transition into thecharacterization mode based on the user input.

Furthermore, the A/V hub may transmit one or more frames that includeadditional audio content and playback timing information to theelectronic device, where the playback timing information may specify aplayback time when the electronic device is to playback the additionalaudio content based on the one or more acoustic characteristics.

Another embodiment provides a computer-readable storage medium for usewith the A/V hub. This computer-readable storage medium includes aprogram module that, when executed by the A/V hub, cause the A/V hub toperform at least some of the aforementioned operations.

Another embodiment provides a method for selectively determining one ormore acoustic characteristics of the environment that includes the A/Vhub. This method includes at least some of the operations performed bythe A/V hub.

Another embodiment provides the electronic device.

A third group of described embodiments includes an audio/video (A/V)hub. This A/V hub includes: one or more acoustic transducers that,during operation, measure sound output by electronic devices in anenvironment that includes the A/V hub and the electronic devices; one ormore antennas; and an interface circuit that, during operation,communicates with the electronic devices using wireless communication.During operation, the A/V hub measures the sound output by theelectronic devices using the one or more acoustic transducers, where thesound corresponds to one or more acoustic-characterization patterns.Then, the A/V hub calculates current time offsets between clocks in theelectronic devices and a clock in the A/V hub based on the measuredsound, one or more times when the electronic devices output the soundand the one or more acoustic-characterization patterns. Next, the A/Vhub transmits, using wireless communication, one or more frames thatinclude audio content and playback timing information to the electronicdevices, where the playback timing information specifies playback timeswhen the electronic devices are to playback the audio content based onthe current time offsets. Moreover, the playback times of the electronicdevices have a temporal relationship so that the playback of the audiocontent by the electronic devices is coordinated.

Note that the measured sound may include information that specifies theone or more times when the electronic devices output the sound, and theone or more times may correspond to the clocks in the electronicdevices.

Moreover, the A/V hub may provide to the electronic devices, via thewireless communication, one or more times when the electronic devicesare to output the sound, and the one or more times may correspond to theclock in the A/V hub.

Furthermore, a given electronic device may output the sound at adifferent time in the one or more times than those used by a remainderof the electronic devices. Alternatively or additionally, the soundoutput by a given electronic device may correspond to a givenacoustic-characterization patterns, which may be different from thoseused by the remainder of the electronic devices.

Note that the acoustic-characterization patterns may include pulses.Moreover, the sound may be in a range of frequencies outside of humanhearing.

In some embodiments, the A/V hub modifies the measured sound based on anacoustic transfer function of the environment in at least a band offrequencies.

Moreover, the temporal relationship may have a non-zero value, so thatat least some of the electronic devices are instructed to playback theaudio content with a phase relative to each other by using differentvalues of the playback times. For example, the different playback timesmay be based on: acoustic characterization of the environment; a desiredacoustic characteristic in the environment; and/or an estimated locationof a listener relative to the electronic devices.

Another embodiment provides a computer-readable storage medium for usewith the A/V hub. This computer-readable storage medium includes aprogram module that, when executed by the A/V hub, cause the A/V hub toperform at least some of the aforementioned operations.

Another embodiment provides a method for coordinating playback of audiocontent. This method includes at least some of the operations performedby the A/V hub.

Another embodiment provides one or more of the electronic devices.

A fourth group of described embodiments includes an audio/video (A/V)hub. This A/V hub includes: one or more antennas; and an interfacecircuit that, during operation, communicates with electronic devicesusing wireless communication. During operation, the A/V hub calculatesan estimated location of a listener relative to the electronic devicesin an environment that includes the A/V hub and the electronic devices.Then, the A/V hub transmits one or more frames that include audiocontent and playback timing information to the electronic devices, wherethe playback timing information specifies playback times when theelectronic devices are to playback the audio content based on theestimated location. Note that the playback times of the electronicdevices have a temporal relationship so that the playback of the audiocontent by the electronic devices is coordinated.

Moreover, the interface circuit may communicate with another electronicdevice, and the estimated location of the listener may be calculatedbased on the communication with the other electronic device.Furthermore, the A/V hub may include an acoustic transducer thatperforms sound measurements in the environment, and the estimatedlocation of the listener may be calculated based on the soundmeasurements. Alternatively or additionally, the interface circuit maycommunicate with other electronic devices in the environment and mayreceive additional sound measurements of the environment from the otherelectronic devices, and the estimated location of the listener may becalculated based on the additional sound measurements. In someembodiments, the interface circuit performs time-of-flight measurements,and the estimated location of the listener is calculated based on thetime-of-flight measurements.

Note that the playback times may be based on current time offsetsbetween clocks in the electronic devices and a clock in the A/V hub.

Moreover, the A/V hub may calculate additional estimated locations ofadditional listeners relative to the electronic devices in theenvironment, and the playback times may be based on the estimatedlocation and the additional estimated locations. For example, theplayback times may be based on an average of the estimated location andthe additional estimated locations. Alternatively, the playback timesmay be based on a weighted average of the estimated location and theadditional estimated locations.

Furthermore, the temporal relationship may have a non-zero value, sothat at least some of the electronic devices are instructed to playbackthe audio content with a phase relative to each other by using differentvalues of the playback times. In some embodiments, the differentplayback times are based on: acoustic characterization of theenvironment; and/or a desired acoustic characteristic in theenvironment.

Another embodiment provides a computer-readable storage medium for usewith the A/V hub. This computer-readable storage medium includes aprogram module that, when executed by the A/V hub, cause the A/V hub toperform at least some of the aforementioned operations.

Another embodiment provides a method for calculating an estimatedlocation. This method includes at least some of the operations performedby the A/V hub.

Another embodiment provides one or more of the electronic devices.

A fifth group of described embodiments includes an audio/video (A/V)hub. This A/V hub includes: one or more acoustic transducers that,during operation, measure sound output by electronic devices in anenvironment that includes the A/V hub and the electronic devices; one ormore antennas; and an interface circuit that, during operation,communicates with the electronic devices using wireless communication.During operation, the A/V hub measures the sound output by theelectronic devices using the one or more acoustic transducers, where thesound corresponds to audio content. Then, the A/V hub aggregates theelectronic devices into two or more subsets based on the measured sound.Moreover, the A/V hub determines playback timing information for thesubsets, where the playback timing information specifies playback timeswhen the electronic devices in a given subset are to playback the audiocontent. Next, the A/V hub transmits, using wireless communication, oneor more frames that include the audio content and playback timinginformation to the electronic devices, where the playback times of theelectronic devices in at least the given subset have a temporalrelationship so that the playback of the audio content by the electronicdevices in the given subset is coordinated.

Note that the different subsets may be located in different rooms in theenvironment.

Moreover, at least one of the subsets may playback different audiocontent than a remainder of the subsets.

Furthermore, the aggregation of the electronic devices into the two ormore subsets may be based on: the different audio content; an acousticdelay of the measured sound; and/or a desired acoustic characteristic inthe environment.

Additionally, the A/V hub may calculate an estimated location of alistener relative to the electronic devices, and the aggregation of theelectronic devices into the two or more subsets may be based on theestimated location of the listener.

In some embodiments, the A/V hub modifies the measured sound based on anacoustic transfer function of the environment in at least a band offrequencies.

Moreover, the A/V hub may determine playback volumes for the subsetsthat are used when the subsets playback the audio content, and the oneor more frames may include information that specifies the playbackvolumes. For example, a playback volume for at least one of the subsetsmay be different than the playback volumes of a remainder of thesubsets. Alternatively or additionally, the playback volumes may reduceacoustic cross-talk among the two or more subsets.

Another embodiment provides a computer-readable storage medium for usewith the A/V hub. This computer-readable storage medium includes aprogram module that, when executed by the A/V hub, cause the A/V hub toperform at least some of the aforementioned operations.

Another embodiment provides a method for aggregating electronic devices.This method includes at least some of the operations performed by theA/V hub.

Another embodiment provides one or more of the electronic devices.

A sixth group of described embodiments includes an audio/video (A/V)hub. This A/V hub includes: one or more acoustic transducers that,during operation, measure sound output by electronic devices in anenvironment that includes the A/V hub and the electronic devices; one ormore antennas; and an interface circuit that, during operation,communicates with the electronic devices using wireless communication.During operation, the A/V hub measures the sound output by theelectronic devices using the one or more acoustic transducers, where thesound corresponds to audio content. Then, the A/V hub compares themeasured sound to a desired acoustic characteristic at a first locationin the environment based on the first location, a second location of theA/V hub, and an acoustic transfer function of the environment in atleast a band of frequencies, where the comparison involves calculatingthe acoustic transfer function at the first location based on theacoustic transfer function at other locations in the environment andcorrecting the measured sound based on the calculated the acoustictransfer function at the first location. Moreover, the A/V hubdetermines equalized audio content based on the comparison and the audiocontent. Next, the A/V hub transmits, using wireless communication, oneor more frames that include the equalized audio content to theelectronic devices to facilitate output by the electronic devices ofadditional sound, which corresponds to the equalized audio content.

Note that the first location may include an estimated location of alistener relative to the electronic devices, and the A/V hub maycalculate the estimated location of the listener. For example, the A/Vhub may calculate the estimated location of the listener based on thesound measurements. Alternatively or additionally, the interface circuitmay: communicate with another electronic device; and may calculate theestimated location of the listener based on the communication with theother electronic device. In particular, the communication with the otherelectronic device may include wireless ranging, and the estimatedlocation may be calculated based on the wireless ranging and an angle ofarrival of wireless signals from the other electronic device. In someembodiments, the interface circuit: performs time-of-flightmeasurements; and calculates the estimated location of the listenerbased on the time-of-flight measurements.

Moreover, the interface circuit may communicate with other electronicdevices in the environment and may receive additional sound measurementsof the environment from the other electronic devices. Then, the A/V hubmay perform one or more additional comparisons of the additional soundmeasurements to the desired acoustic characteristic at the firstlocation in the environment based on one or more third locations of theother electronic devices and the acoustic transfer function of theenvironment in at least a band of frequencies, and the equalized audiocontent is further determined based on the one or more additionalcomparisons. Furthermore, the interface circuit may determine the one ormore third locations based on the communication with the otherelectronic devices. For example, the communication with the otherelectronic devices may include wireless ranging, and the one or morethird locations may be calculated based on the wireless ranging andangles of arrival of wireless signals from the other electronic devices.Alternatively or additionally, the interface circuit may receiveinformation specifying the third locations from the other electronicdevices.

In some embodiments, the desired acoustic characteristic is based on atype of audio playback, which may include: monophonic, stereophonicand/or multichannel.

Moreover, the A/V hub may determine playback timing information thatspecifies playback times when the electronic devices playback theequalized audio content, the one or more frames further may include theplayback timing information, and the playback times of the electronicdevices have a temporal relationship so that the playback of the audiocontent by the electronic devices is coordinated.

Another embodiment provides a computer-readable storage medium for usewith the A/V hub. This computer-readable storage medium includes aprogram module that, when executed by the A/V hub, cause the A/V hub toperform at least some of the aforementioned operations.

Another embodiment provides a method for determining the equalized audiocontent. This method includes at least some of the operations performedby the A/V hub.

Another embodiment provides one or more of the electronic devices.

A seventh group of described embodiments includes an audio/video (A/V)hub. This A/V hub includes: one or more antennas; and an interfacecircuit that, during operation, communicates with electronic devicesusing wireless communication. During operation, the A/V hub receives,via the wireless communication, frames from the electronic devices.Then, the A/V hub stores receive times when the frames were received,where the receive times are based on a clock in the A/V hub. Moreover,the A/V hub calculates current time offsets between clocks in theelectronic devices and the clock in the A/V hub based on the receivetimes and expected transmit times of the frames, where the expectedtransmit times are based on coordination of the clocks in the electronicdevices and the clock in the A/V hub at a previous time and a predefinedtransmit schedule of the frames. Next, the A/V hub transmits one or moreframes that include audio content and playback timing information to theelectronic devices, where the playback timing information specifiesplayback times when the electronic devices are to playback the audiocontent based on the current time offsets. Furthermore, the playbacktimes of the electronic devices have a temporal relationship so that theplayback of the audio content by the electronic devices is coordinated.

Note that the temporal relationship may have a non-zero value, so thatat least some of the electronic devices are instructed to playback theaudio content with a phase relative to each other by using differentvalues of the playback times. For example, the different playback timesmay be based on acoustic characterization of an environment thatincludes the electronic devices and the A/V hub. Alternatively oradditionally, the different playback times may be based on a desiredacoustic characteristic in the environment.

In some embodiments, the electronic devices are located at vectordistances from the A/V hub, and the interface circuit determinesmagnitudes of the vector distances based on transmit times of the framesand the receive times using wireless ranging. Moreover, the interfacecircuit may determine angles of the vector distances based on the angleof arrival of wireless signals associated with the frames that arereceived by the one or more antennas during the wireless communication.Furthermore, the different playback times may be based on the determinedvector distances.

Alternatively or additionally, the different playback times are based onan estimated location of a listener relative to the electronic devices.For example, the interface circuit may: communicate with anotherelectronic device; and calculate the estimated location of the listenerbased on the communication with the other electronic device. Moreover,the A/V hub may include an acoustic transducer that performs soundmeasurements of the environment that includes the A/V hub, and the A/Vhub may calculate the estimated location of the listener based on thesound measurements. Furthermore, the interface circuit may communicatewith other electronic devices in the environment and may receiveadditional sound measurements of the environment from the otherelectronic devices. In these embodiments, the A/V hub calculates theestimated location of the listener based on the additional soundmeasurements. In some embodiments, the interface circuit: performstime-of-flight measurements; and calculates the estimated location ofthe listener based on the time-of-flight measurements.

Note that the coordination of the clocks in the electronic devices andthe clock in the A/V hub may have occurred during an initialization modeof operation.

Moreover, the current time offsets may be based on models of clock driftin the electronic devices.

Another embodiment provides a computer-readable storage medium for usewith the A/V hub. This computer-readable storage medium includes aprogram module that, when executed by the A/V hub, cause the A/V hub toperform at least some of the aforementioned operations.

Another embodiment provides a method for coordinating playback of audiocontent. This method includes at least some of the operations performedby the A/V hub.

Another embodiment provides one or more of the electronic devices.

This Summary is only provided for purposes of illustrating someexemplary embodiments, so as to provide a basic understanding of someaspects of the subject matter described herein. Accordingly, it will beappreciated that the above-described features are only examples andshould not be construed to narrow the scope or spirit of the subjectmatter described herein in any way. Other features, aspects, andadvantages of the subject matter described herein will become apparentfrom the following Detailed Description, Figures, and Claims.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a block diagram illustrating a system with electronic devicesin accordance with an embodiment of the present disclosure.

FIG. 2 is a flow diagram illustrating a method for coordinating playbackof audio content in accordance with an embodiment of the presentdisclosure.

FIG. 3 is a drawing illustrating communication among the electronicdevices in FIG. 1 in accordance with an embodiment of the presentdisclosure.

FIG. 4 is a drawing illustrating coordinating playback of audio contentby the electronic devices in FIG. 1 in accordance with an embodiment ofthe present disclosure.

FIG. 5 is a flow diagram illustrating a method for coordinating playbackof audio content in accordance with an embodiment of the presentdisclosure.

FIG. 6 is a drawing illustrating communication among the electronicdevices in FIG. 1 in accordance with an embodiment of the presentdisclosure.

FIG. 7 is a drawing illustrating coordinating playback of audio contentby the electronic devices in FIG. 1 in accordance with an embodiment ofthe present disclosure.

FIG. 8 is a flow diagram illustrating a method for coordinating playbackof audio content in accordance with an embodiment of the presentdisclosure.

FIG. 9 is a drawing illustrating communication among the electronicdevices in FIG. 1 in accordance with an embodiment of the presentdisclosure.

FIG. 10 is a drawing illustrating coordinating playback of audio contentby the electronic devices in FIG. 1 in accordance with an embodiment ofthe present disclosure.

FIG. 11 is a flow diagram illustrating a method for selectivelydetermining one or more acoustic characteristics of an environment inaccordance with an embodiment of the present disclosure.

FIG. 12 is a drawing illustrating communication among the electronicdevices in FIG. 1 in accordance with an embodiment of the presentdisclosure.

FIG. 13 is a drawing illustrating selective acoustic characterization ofan environment that includes the electronic devices in FIG. 1 inaccordance with an embodiment of the present disclosure.

FIG. 14 is a flow diagram illustrating a method for calculating anestimated location in accordance with an embodiment of the presentdisclosure.

FIG. 15 is a drawing illustrating communication among the electronicdevices in FIG. 1 in accordance with an embodiment of the presentdisclosure.

FIG. 16 is a drawing illustrating calculating an estimated location ofone or more listeners relative to the electronic devices in FIG. 1 inaccordance with an embodiment of the present disclosure.

FIG. 17 is a flow diagram illustrating a method for aggregatingelectronic devices in accordance with an embodiment of the presentdisclosure.

FIG. 18 is a drawing illustrating communication among the electronicdevices in FIG. 1 in accordance with an embodiment of the presentdisclosure.

FIG. 19 is a drawing illustrating aggregating the electronic devices inFIG. 1 in accordance with an embodiment of the present disclosure.

FIG. 20 is a flow diagram illustrating a method for determiningequalized audio content in accordance with an embodiment of the presentdisclosure.

FIG. 21 is a drawing illustrating communication among the electronicdevices in FIG. 1 in accordance with an embodiment of the presentdisclosure.

FIG. 22 is a drawing illustrating determining equalized audio contentusing the electronic devices in FIG. 1 in accordance with an embodimentof the present disclosure.

FIG. 23 is a block diagram illustrating one of the electronic devices ofFIG. 1 in accordance with an embodiment of the present disclosure.

Note that like reference numerals refer to corresponding partsthroughout the drawings. Moreover, multiple instances of the same partare designated by a common prefix separated from an instance number by adash.

DETAILED DESCRIPTION

In a first group of embodiments, an audio/video (A/V) hub thatcoordinates playback of audio content is described. In particular, theA/V hub may calculate current time offsets between clocks in electronicdevices (such as electronic devices that include speakers) and a clockin the A/V hub based on differences between transmit times of framesfrom the electronic devices and receive times when the frames werereceived. For example, the current time offsets may be calculated usingwireless ranging by ignoring distances between the A/V hub and theelectronic devices. Then, the A/V hub may transmit, to the electronicdevices, one or more frames that include audio content and playbacktiming information, which may specify playback times when the electronicdevices are to playback the audio content based on the current timeoffsets. Furthermore, the playback times of the electronic devices mayhave a temporal relationship so that the playback of the audio contentby the electronic devices is coordinated.

By coordinating the playback of the audio content by the electronicdevices, this coordination technique may provide an improved acousticexperience in an environment that includes the A/V hub and theelectronic devices. For example, the coordination technique may correctfor clock drift between the A/V hub and the electronic devices.Alternatively or additionally, the coordination technique may correct oradapt for acoustic characteristics of the environment and/or based on adesired acoustic characteristic in the environment. In addition, thecoordination technique may correct the playback times based on anestimated location of a listener relative to the electronic devices. Inthese ways, the coordination technique may improve the acoustic qualityand, more generally, the user experience when using the A/V hub and theelectronic devices. Consequently, the coordination technique mayincrease customer loyalty and revenue of a provider of the A/V hub andthe electronic devices.

In a second group of embodiments, an audio/video (A/V) hub thatselectively determines one or more acoustic characteristics of anenvironment that includes the A/V hub is described. In particular, theA/V hub may detect, using wireless communication, an electronic device(such as an electronic device that includes a speaker) in theenvironment. Then, the A/V hub may determine a change condition, such aswhen the electronic device was not previously detected in theenvironment and/or a change in a location of the electronic device. Inresponse to determining the change condition, the A/V hub may transitioninto a characterization mode. During the characterization mode, the A/Vhub may: provide instructions to the electronic device to playback audiocontent at a specified playback time; determine one or more acousticcharacteristics of the environment based on acoustic measurements in theenvironment; and store the one or more acoustic characteristics and/or alocation of the electronic device in memory.

By selectively determining the one or more acoustic characteristics,this characterization technique may facilitate an improved acousticexperience in the environment that includes the A/V hub and theelectronic device. For example, the characterization technique mayidentify the changes and characterize the modified environment, whichmay be subsequently used to correct for the impact of the change duringplayback of audio content by one or more electronic devices (includingthe electronic device). In these ways, the characterization techniquemay improve acoustic quality and, more generally, the user experiencewhen using the A/V hub and the electronic devices. Consequently, thecharacterization technique may increase customer loyalty and revenue ofa provider of the A/V hub and the electronic devices.

In a third group of embodiments, an audio/video (A/V) hub thatcoordinates playback of audio content is described. In particular, theA/V hub may calculate current time offsets between clocks in electronicdevices (such as electronic devices that include speakers) and a clockin the A/V hub based on measured sound corresponding to one or moreacoustic-characterization patterns, one or more times when theelectronic devices output the sound and the one or moreacoustic-characterization patterns. Then, the A/V hub may transmit, tothe electronic devices, one or more frames that include audio contentand playback timing information, which may specify playback times whenthe electronic devices are to playback the audio content based on thecurrent time offsets. Moreover, the playback times of the electronicdevices may have a temporal relationship so that the playback of theaudio content by the electronic devices is coordinated.

By coordinating the playback of the audio content by the electronicdevices, this coordination technique may provide an improved acousticexperience in an environment that includes the A/V hub and theelectronic devices. For example, the coordination technique may correctfor clock drift between the A/V hub and the electronic devices.Alternatively or additionally, the coordination technique may correct oradapt for acoustic characteristics of the environment and/or based on adesired acoustic characteristic in the environment. In addition, thecoordination technique may correct the playback times based on anestimated location of a listener relative to the electronic devices. Inthese ways, the coordination technique may improve the acoustic qualityand, more generally, the user experience when using the A/V hub and theelectronic devices. Consequently, the coordination technique mayincrease customer loyalty and revenue of a provider of the A/V hub andthe electronic devices.

In a fourth group of embodiments, an audio/video (A/V) hub thatcalculates an estimated location is described. In particular, the A/Vhub may calculate an estimated location of a listener relative toelectronic devices (such as electronic devices that include speakers) inan environment that includes the A/V hub and the electronic devicesbased on: communication with another electronic device; soundmeasurements in the environment; and/or time-of-flight measurements.Then, the A/V hub may transmit, to the electronic devices, one or moreframes that include audio content and playback timing information, whichmay specify playback times when the electronic devices are to playbackthe audio content based on the estimated location. Moreover, theplayback times of the electronic devices may have a temporalrelationship so that the playback of the audio content by the electronicdevices is coordinated.

By calculating the estimated location of the listener, thischaracterization technique may facilitate an improved acousticexperience in the environment that includes the A/V hub and theelectronic devices. For example, the characterization technique maytrack changes in the location of the listener in the environment, whichmay be subsequently used to correct or adapt playback of audio contentby one or more electronic devices. In these ways, the characterizationtechnique may improve the acoustic quality and, more generally, the userexperience when using the A/V hub and the electronic devices.Consequently, the characterization technique may increase customerloyalty and revenue of a provider of the A/V hub and the electronicdevices.

In a fifth group of embodiments, an audio/video (A/V) hub thataggregates electronic devices is described. In particular, the A/V hubmay measure sound, corresponding to audio content, output by electronicdevices (such as electronic devices that include speakers). Then, theA/V hub may aggregate the electronic devices into two or more subsetsbased on the measured sound. Moreover, the A/V hub may determine, forthe subsets, playback timing information, which may specify playbacktimes when the electronic devices in a given subset are to playback theaudio content. Next, the A/V hub may transmit, to the electronicdevices, one or more frames that include the audio content and playbacktiming information, where the playback times of the electronic devicesin at least the given subset have a temporal relationship so that theplayback of the audio content by the electronic devices in the givensubset is coordinated.

By aggregating the electronic devices, this characterization techniquemay facilitate an improved acoustic experience in the environment thatincludes the A/V hub and the electronic devices. For example, thecharacterization technique may aggregate the electronic devices basedon: different audio content; an acoustic delay of the measured sound;and/or a desired acoustic characteristic in the environment. Inaddition, the A/V hub may determine playback volumes for the subsetsthat are used when the subsets playback the audio content in order toreduce acoustic cross-talk among the two or more subsets. In these ways,the characterization technique may improve the acoustic quality and,more generally, the user experience when using the A/V hub and theelectronic devices. Consequently, the characterization technique mayincrease customer loyalty and revenue of a provider of the A/V hub andthe electronic devices.

In a sixth group of embodiments, an audio/video (A/V) hub thatdetermines equalized audio content is described. In particular, the A/Vhub may measure the sound, corresponding to audio content, output byelectronic devices (such as electronic devices that include speakers).Then, the A/V hub may compare the measured sound to a desired acousticcharacteristic at a first location in the environment based on the firstlocation, a second location of the A/V hub, and an acoustic transferfunction of the environment in at least a band of frequencies. Forexample, the comparison may involve calculating the acoustic transferfunction at the first location based on the acoustic transfer functionat other locations in the environment and correcting the measured soundbased on the calculated the acoustic transfer function at the firstlocation. Moreover, the A/V hub may determine the equalized audiocontent based on the comparison and the audio content. Next, the A/V hubmay transmit, to the electronic devices, one or more frames that includethe equalized audio content to facilitate output by the electronicdevices of additional sound, which corresponds to the equalized audiocontent.

By determining the equalized audio content, this signal-processingtechnique may facilitate an improved acoustic experience in theenvironment that includes the A/V hub and the electronic devices. Forexample, the signal-processing may dynamically modify the audio contentbased on an estimated location of a listener relative to locations ofthe electronic devices and the acoustic transfer function of theenvironment in at least the band of frequencies. This may allow adesired acoustic characteristic or a type of audio playback (such asmonophonic, stereophonic or multichannel) to be achieved at theestimated location in the environment. In these ways, thesignal-processing technique may improve the acoustic quality and, moregenerally, the user experience when using the A/V hub and the electronicdevices. Consequently, the signal-processing technique may increasecustomer loyalty and revenue of a provider of the A/V hub and theelectronic devices.

In a seventh group of embodiments, an audio/video (A/V) hub thatcoordinates playback of audio content is described. In particular, theA/V hub may calculate current time offsets between clocks in electronicdevices (such as electronic devices that include speakers) and a clockin the A/V hub based on differences between receive times when framesare received from electronic devices and expected transmit times of theframes. For example, the expected transmit times may be based oncoordination of clocks in the electronic devices and a clock in the A/Vhub at a previous time and a predefined transmit schedule of the frames.Then, the A/V hub may transmit, to the electronic devices, one or moreframes that include audio content and playback timing information, whichmay specify playback times when the electronic devices are to playbackthe audio content based on the current time offsets. Furthermore, theplayback times of the electronic devices may have a temporalrelationship so that the playback of the audio content by the electronicdevices is coordinated.

By coordinating the playback of the audio content by the electronicdevices, this coordination technique may provide an improved acousticexperience in an environment that includes the A/V hub and theelectronic devices. For example, the coordination technique may correctfor clock drift between the A/V hub and the electronic devices.Alternatively or additionally, the coordination technique may correct oradapt for acoustic characteristics of the environment and/or based on adesired (or target) acoustic characteristic in the environment. Inaddition, the coordination technique may correct the playback timesbased on an estimated location of a listener relative to the electronicdevices. In these ways, the coordination technique may improve theacoustic quality and, more generally, the user experience when using theA/V hub and the electronic devices. Consequently, the coordinationtechnique may increase customer loyalty and revenue of a provider of theA/V hub and the electronic devices.

In the discussion that follows, the A/V hub (which is sometimes referredto as ‘a coordination device’), an A/V display device, a portableelectronic device, one or more receiver devices, and/or one or moreelectronic devices (such as a speaker and, more generally, aconsumer-electronic device) may include radios that communicate packetsor frames in accordance with one or more communication protocols, suchas: an Institute of Electrical and Electronics Engineers (IEEE) 802.11standard (which is sometimes referred to as ‘Wi-Fi®,’ from the Wi-Fi®Alliance of Austin, Tex.), Bluetooth® (from the Bluetooth SpecialInterest Group of Kirkland, Wash.), a cellular-telephone communicationprotocol, a near-field-communication standard or specification (from theNFC Forum of Wakefield, Mass.), and/or another type of wirelessinterface. For example, the cellular-telephone communication protocolmay include or may be compatible with: a 2^(nd) generation of mobiletelecommunication technology, a 3^(rd) generation of mobiletelecommunications technology (such as a communication protocol thatcomplies with the International Mobile Telecommunications-2000specifications by the International Telecommunication Union of Geneva,Switzerland), a 4^(th) generation of mobile telecommunicationstechnology (such as a communication protocol that complies with theInternational Mobile Telecommunications Advanced specification by theInternational Telecommunication Union of Geneva, Switzerland), and/oranother cellular-telephone communication technique. In some embodiments,the communication protocol includes Long Term Evolution or LTE. However,a wide variety of communication protocols may be used (such asEthernet). In addition, the communication may occur via a wide varietyof frequency bands. Note that the portable electronic device, the A/Vhub, the A/V display device, and/or the one or more electronic devicesmay communicate using infra-red communication that is compatible with aninfra-red communication standard (including unidirectional orbidirectional infra-red communication).

Moreover, A/V content in following discussion may include video andassociated audio (such as music, sound, dialog, etc.), video only oraudio only.

Communication among electronic devices is shown in FIG. 1, whichpresents a block diagram illustrating a system 100 with a portableelectronic device 110 (such as a remote control or a cellulartelephone), one or more A/Vhubs (such as A/V hub 112), one or more A/Vdisplay devices 114 (such as a television, a monitor, a computer and,more generally, a display associated with an electronic device), one ormore receiver devices (such as receiver device 116, e.g., a localwireless receiver associated with a proximate A/V display device 114-1that can receive frame-by-frame transcoded A/V content from A/V hub 112for display on A/V display device 114-1), one or more speakers 118 (and,more generally, one or more electronic devices that include one or morespeakers) and/or one or more content sources 120 associated with one ormore content providers (e.g., a radio receiver, a video player, asatellite receiver, an access point that provides a connection to awired network such as the Internet, a media or a content source, aconsumer-electronic device, an entertainment device, a set-top box,over-the-top content delivered over the Internet or a network withoutinvolvement of a cable, satellite or multiple-system operator, asecurity camera, a monitoring camera, etc.). Note that A/V hub 112, A/Vdisplay devices 114, receiver device 116 and speakers 118 are sometimescollectively referred to as ‘components’ in system 100. However, A/V hub112, A/V display devices 114, receiver device 116 and/or speakers 118are sometimes referred to as ‘electronic devices.’

In particular, portable electronic device 110 and A/V hub 112 maycommunicate with each other using wireless communication, and one ormore other components in system 100 (such as at least: one of A/Vdisplay devices 114, receiver device 116, one of speakers 118 and/or oneof content sources 120) may communicate using wireless and/or wiredcommunication. During the wireless communication, these electronicdevices may wirelessly communicate while: transmitting advertisingframes on wireless channels, detecting one another by scanning wirelesschannels, establishing connections (for example, by transmittingassociation requests), and/or transmitting and receiving packets orframes (which may include the association requests and/or additionalinformation as payloads, such as information specifying communicationperformance, data, a user interface, A/V content, etc.).

As described further below with reference to FIG. 23, portableelectronic device 110, A/V hub 112, A/V display devices 114, receiverdevice 116, speakers 118 and content sources 120 may include subsystems,such as: a networking subsystem, a memory subsystem and a processorsubsystem. In addition, portable electronic device 110, A/V hub 112,receiver device 116, and/or speakers 118, and optionally one or more ofA/V display devices 114 and/or content sources 120, may include radios122 in the networking subsystems. For example, a radio or receiverdevice may be in an A/V display device, e.g., radio 122-5 is included inA/V display device 114-2.) Moreover, note that radios 122 may beinstances of the same radio or may be different from each other. Moregenerally, portable electronic device 110, A/V hub 112, receiver device116 and/or speakers 118 (and optionally A/V display devices 114 and/orcontent sources 120) can include (or can be included within) anyelectronic devices with the networking subsystems that enable portableelectronic device 110, A/V hub 112 receiver device 116 and/or speakers118 (and optionally A/V display devices 114 and/or content sources 120)to wirelessly communicate with each other. This wireless communicationcan comprise transmitting advertisements on wireless channels to enableelectronic devices to make initial contact or detect each other,followed by exchanging subsequent data/management frames (such asassociation requests and responses) to establish a connection, configuresecurity options (e.g., Internet Protocol Security), transmit andreceive packets or frames via the connection, etc.

As can be seen in FIG. 1, wireless signals 124 (represented by a jaggedline) are transmitted from radio 122-1 in portable electronic device110. These wireless signals may be received by at least one of: A/V hub112, receiver device 116 and/or at least one of speakers 118 (and,optionally, one or more of A/V display devices 114 and/or contentsources 120). For example, portable electronic device 110 may transmitpackets. In turn, these packets may be received by a radio 122-2 in A/Vhub 112. This may allow portable electronic device 110 to communicateinformation to A/V hub 112. While FIG. 1 illustrates portable electronicdevice 110 transmitting packets, note that portable electronic device110 may also receive packets from A/V hub 112 and/or one or more othercomponents in system 100. More generally, wireless signals may betransmitted and/or received by one or more of the components in system100.

In the described embodiments, processing of a packet or frame inportable electronic device 110, A/V hub 112, receiver device 116 and/orspeakers 118 (and optionally one or more of A/V display devices 114and/or content sources 120) includes: receiving wireless signals 124with the packet or frame; decoding/extracting the packet or frame fromreceived wireless signals 124 to acquire the packet or frame; andprocessing the packet or frame to determine information contained in thepacket or frame (such as the information associated with a data stream).For example, the information from portable electronic device 110 mayinclude user-interface activity information associated with a userinterface displayed on touch-sensitive display (TSD) 128 in portableelectronic device 110, which a user of portable electronic device 110uses to control at least: A/V hub 112, at least one of A/V displaydevices 114, at least one of speakers 118 and/or at least one of contentsources 120. (In some embodiments, instead of or in additional totouch-sensitive display 128, portable electronic device 110 includes auser interface with physical knobs and/or buttons that a user can use tocontrol at least: A/V hub 112 one of A/V display devices 114, at leastone of speakers 118 and/or one of content sources 120.) Alternatively,the information from portable electronic device 110, A/V hub 112, one ormore of A/V display devices 114, receiver device 116, one or more ofspeakers 118 and/or one or more of content sources 120 may specifycommunication performance about the communication between portableelectronic device 110 and one or more other components in system 100.Moreover, the information from A/V hub 112 may include device-stateinformation about a current device state of at least one of A/V displaydevices 114, at least one of speakers 118 and/or one of content sources120 (such as on, off, play, rewind, fast forward, a selected channel,selected A/V content, a content source, etc.), or may includeuser-interface information for the user interface (which may bedynamically updated based on the device-state information and/or theuser-interface activity information). Furthermore, the information fromat least A/V hub 112 and/or one of content sources 120 may include audioand/or video (which is sometimes denoted as ‘audio/video’ or ‘A/V’content) that are displayed or presented on one or more of A/V displaydevices 114, as well as display instructions that specify how the audioand/or video are to be displayed or presented.

However, as noted previously, the audio and/or video may be communicatedbetween components in system 100 via wired communication. Therefore, asshown in FIG. 1, there may be a wired cable or link, such as ahigh-definition multimedia-interface (HDMI) cable 126, such as betweenA/V hub 112 and A/V display device 114-3. While the audio and/or videomay be included in or associated with HDMI content, in other embodimentsthe audio content may be included in or associated with A/V content thatis compatible with another format or standard is used in the embodimentsof the disclosed communication technique. For example, the A/V contentmay include or may be compatible with: H.264, MPEG-2, a QuickTime videoformat, MPEG-4, MP4, and/or TCP/IP. Moreover, the video mode of the A/Vcontent may be 720p, 1080i, 1080p, 1440p, 2000, 2160p, 2540p, 4000pand/or 4320p.

Note that A/V hub 112 may determine display instructions (with a displaylayout) for the A/V content based on a format of a display in one of A/Vdisplay devices 114, such as A/V display device 114-1. Alternatively,A/V hub 112 can use pre-determined display instructions or A/V hub 112can modify or transform the A/V content based on the display layout sothat the modified or transformed A/V content has an appropriate formatfor display on the display. Moreover, the display instructions mayspecify information to be displayed on the display in A/V display device114-1, including where A/V content is displayed (such as in a centralwindow, in a tiled window, etc.). Consequently, the information to bedisplayed (i.e., an instance of the display instructions) may be basedon a format of the display, such as: a display size, display resolution,display aspect ratio, display contrast ratio, a display type, etc.Furthermore, note that when A/V hub 112 receives the A/V content fromone of content sources 120, A/V hub 112 may provide the A/V content anddisplay instructions to A/V display device 114-1 as frames with the A/Vcontent are received from one of content sources 120 (e.g., in realtime), so that the A/V content is displayed on the display in A/Vdisplay device 114-1. For example, A/V hub 112 may collect the A/Vcontent in a buffer until a frame is received, and then A/V hub 112 mayprovide the complete frame to A/V display device 114-1. Alternatively,A/V hub 112 may provide packets with portions of a frame to A/V displaydevice 114-1 as they are received. In some embodiments, the displayinstructions may be provided to A/V display device 114-1 differentially(such as when the display instructions change), regularly orperiodically (such as in one of every N packets or in a packet in eachframe) or in each packet.

Moreover, note that the communication between portable electronic device110, A/V hub 112, one or more of A/V display devices 114, receiverdevice 116, one or more of speakers 118 and/or one or more contentsources 120 may be characterized by a variety of performance metrics,such as: a received signal strength indicator (RSSI), a data rate, adata rate discounting radio protocol overhead (which is sometimesreferred to as a ‘throughput’), an error rate (such as a packet errorrate, or a retry or resend rate), a mean-square error of equalizedsignals relative to an equalization target, intersymbol interference,multipath interference, a signal-to-noise ratio, a width of an eyepattern, a ratio of number of bytes successfully communicated during atime interval (such as 1-10 s) to an estimated maximum number of bytesthat can be communicated in the time interval (the latter of which issometimes referred to as the ‘capacity’ of a channel or link), and/or aratio of an actual data rate to an estimated maximum data rate (which issometimes referred to as ‘utilization’). Moreover, the performanceduring the communication associated with different channels may bemonitored individually or jointly (e.g., to identify dropped packets).

The communication between portable electronic device 110, A/V hub 112,one of A/V display devices 114, receiver device 116 one of speakers 118and/or one or more of content sources 120 in FIG. 1 may involve one ormore independent, concurrent data streams in different wireless channels(or even different communication protocols, such as different Wi-Ficommunication protocols) in one or more connections or links, which maybe communicated using multiple radios. Note that the one or moreconnections or links may each have a separate or different identifier(such as a different service set identifier) on a wireless network insystem 100 (which may be a proprietary network or a public network).Moreover, the one or more concurrent data streams may, on a dynamic orpacket-by-packet basis, be partially or completely redundant to improveor maintain the performance metrics even when there are transientchanges (such as interference, changes in the amount of information thatneeds to be communicated, movement of portable electronic device 110,etc.), and to facilitate services (while remaining compatible with thecommunication protocol, e.g., a Wi-Fi communication protocol) such as:channel calibration, determining of one or more performance metrics,performing quality-of-service characterization without disrupting thecommunication (such as performing channel estimation, determining linkquality, performing channel calibration and/or performing spectralanalysis associated with at least one channel), seamless handoff betweendifferent wireless channels, coordinated communication betweencomponents, etc. These features may reduce the number of packets thatare resent, and, thus, may decrease the latency and avoid disruption ofthe communication and may enhance the experience of one or more usersthat are viewing A/V content on one or more of A/V display devices 114and/or listening to audio output by one or more of speakers 118.

As noted previously, a user may control at least A/V hub 112, at leastone of A/V display devices 114, at least one of speakers 118 and/or atleast one of content sources 120 via the user interface displayed ontouch-sensitive display 128 on portable electronic device 110. Inparticular, at a given time, the user interface may include one or morevirtual icons that allow the user to activate, deactivate or changefunctionality or capabilities of at least: A/V hub 112, at least one ofA/V display devices 114, at least one of speakers 118 and/or at leastone of content sources 120. For example, a given virtual icon in theuser interface may have an associated strike area on a surface oftouch-sensitive display 128. If the user makes and then breaks contactwith the surface (e.g., using one or more fingers or digits, or using astylus) within the strike area, portable electronic device 110 (such asa processor executing a program module) may receive user-interfaceactivity information indicating activation of this command orinstruction from a touch-screen input/output (I/O) controller, which iscoupled to touch-sensitive display 128. (Alternatively, touch-sensitivedisplay 128 may be responsive to pressure. In these embodiments, theuser may maintain contact with touch-sensitive display 128 with anaverage contact pressure that is usually less than a threshold value,such as 10-20 kPa, and may activate a given virtual icon by increase theaverage contact pressure with touch-sensitive display 128 above thethreshold value.) In response, the program module may instruct aninterface circuit in portable electronic device 110 to wirelesslycommunicate the user-interface activity information indicating thecommand or instruction to A/V hub 112, and A/V hub 112 may communicatethe command or the instruction to the target component in system 100(such as A/V display device 114-1). This instruction or command mayresult in A/V display device 114-1 turning on or off, displaying A/Vcontent from a particular content source, performing a trick mode ofoperation (such as fast forward, reverse, fast reverse or skip), etc.For example, A/V hub 112 may request the A/V content from content source120-1, and then may provide the A/V content along with displayinstructions to A/V display device 114-1, so that A/V display device114-1 displays the A/V content. Alternatively or additionally, A/V hub112 may provide audio content associated with video content from contentsource 120-1 to one or more of speakers 118.

As noted previously, it is often challenging to achieve high audioquality in an environment (such as a room, a building, a vehicle, etc.).In particular, achieving high audio quality in the environment typicallyplaces strong constraints on coordination of the loudspeakers, such asspeakers 118. For example, the coordination may need to be maintained to1-5 μs accuracy (which are nonlimiting exemplary values). In someembodiments, the coordination includes synchronization in the timedomain within a temporal or phase accuracy and/or the frequency domainwithin a frequency accuracy. In the absence of suitable coordination,the acoustic quality in the environment may be degraded, with acommensurate impact on listener satisfaction and the overall userexperience when listening to audio content and/or A/V content.

This challenge may be addressed in a coordination technique by directlyor indirectly coordinating speakers 118 with A/V hub 112. As describedbelow with reference to FIGS. 2-4, in some embodiments coordinatedplayback of audio content by speakers 118 may be facilitated usingwireless communication. In particular, because the speed of light isalmost six orders of magnitude faster than the speed of sound, thepropagation delay of wireless signals in an environment (such as a room)is negligible relative to the desired coordination accuracy of speakers118. For example, the desired coordination accuracy of speakers 118 maybe on the order of a microsecond, while the propagation delay in atypical room (e.g., over distances of at most 10-30 m) may be one or twoorders of magnitude smaller. Consequently, techniques such as wirelessranging or radio-based distance measurements may be used to coordinatespeakers 118. In particular, during wireless ranging A/V hub 112 maytransmit a frame or a packet that includes a transmission time and anidentifier of A/V hub 112 based on a clock in A/V hub 112, and a givenone of speakers 118 (such as speaker 118-1) may determine an arrival ora reception time of the frame or packet based on a clock in speaker118-1.

Alternatively, speaker 118-1 may transmit a frame or a packet (which issometimes referred to as an ‘input frame’) that includes a transmissiontime and an identifier of speaker 118-1 based on the clock in speaker118-1, and /V hub 112 may determine an arrival or a reception time ofthe frame or packet based on the clock in /V hub 112. Typically, thedistance between A/V hub 112 and speaker 118-1 is determined based onthe product of the time of fight (the difference of the arrival time andthe transmission time) and the speed of propagation. However, byignoring the physical distance between A/V hub 112 and speaker 118-1,i.e., by assuming instantaneous propagation (which for stationarydevices in the same room or environment introduces a negligible staticoffset), the difference of the arrival time and the transmission timemay dynamically track the drift or the current time offset in thecoordination of the clocks in A/V hub 112 and speaker 118-1 (as well asthe negligible static offset).

The current time offset may be determined by A/V hub 112 or may beprovided to A/V hub 112 by speaker 118-1. Then, A/V hub 112 maytransmit, to speaker 118-1, one or more frames (which are sometimesreferred to as ‘output frames’) that include audio content and playbacktiming information, which may specify playback times when speaker 118-1is to playback the audio content based on the current time offset. Thismay be repeated for other speakers 118. Furthermore, the playback timesof speakers 118 may have a temporal relationship so that the playback ofthe audio content by speakers 118 is coordinated.

In addition to correcting for drift in the clocks, this coordinationtechnique (as well as the other embodiments of the coordinationtechnique described below) may provide an improved acoustic experiencein an environment that includes A/V hub 112 and speakers 118. Forexample, the coordination technique may correct or adapt forpredetermined or dynamically determined acoustic characteristics of theenvironment (as described further below with reference to FIGS. 11-13),based on a desired acoustic characteristic in the environment (such as atype of playback, e.g., monophonic, stereophonic and/or multichannel, anacoustic radiation pattern, such as directed or diffuse,intelligibility, etc.) and/or based on dynamically estimated locationsof one or more listeners relative to speakers 118 (as described furtherbelow with reference to FIGS. 14-16). In addition, the coordinationtechnique may be used in conjunction with dynamic aggregation ofspeakers 118 into groups (as described further below with reference toFIGS. 17-19) and/or with dynamically equalized audio content based audiocontent being played and differences between an acoustic characteristicand the desired acoustic characteristic in the environment (as describedfurther below with reference to FIGS. 20-22).

Note that the wireless ranging (as well as the wireless communication ingeneral) may be performed at or in one or more bands of frequencies,such as at or in: a 2 GHz wireless band, a 5 GHz wireless band, an ISMband, a 60 GHz wireless band, ultra-wide band, etc.

In some embodiments, one or more additional communication techniques maybe used to identify and/or exclude multi-path wireless signals duringthe coordination of speakers 118. For example, A/V hub 112 and/orspeakers 118 may determine the angle of arrival (includingnon-line-of-sight reception) using: a directional antenna, thedifferential time of arrival at an array of antennas with knownlocation(s), and/or the angle of arrival at two receivers having knownlocation (i.e., trilateration or multilateration).

As described further below with reference to FIGS. 5-7, another approachfor coordinating speakers 118 may use scheduled transmission times. Inparticular, during a calibration mode, clocks in A/V hub 112 andspeakers 118 may be coordinated. Subsequently, in a normal operatingmode, A/V hub 112 may transmit frames or packets with an identifier ofA/V hub 112 at predefined transmission times based on the clock in A/Vhub 112. However, because of the relative drift in the clock in A/V hub112, these packets or frames will arrive or be received at speakers 118at different times than the expected predefined transmission times basedon the clocks in speakers 118. Thus, by once again ignoring thepropagation delay, the difference of the arrival time and the predefinedtransmission time of a given frame at a given one of speakers 118 (suchas speaker 118-1) may dynamically track the drift or the current timeoffset in the coordination of the clocks in A/V hub 112 and speaker118-1 (as well as the negligible static offset associated with thepropagation delay).

Alternatively or additionally, after the calibration mode, speakers 118may transmit frames or packets with identifiers of speakers 118 atpredefined transmission times based on the clock in speakers 118.However, because of drift in the clocks in speakers 118, these packetsor frames will arrive or be received by A/V hub 112 at different timesthan the expected predefined transmission times based on the clock inA/V hub 112. Thus, by once again ignoring the propagation delay, thedifference of the arrival time and the predefined transmission time of agiven frame from a given one of speakers 118 (such as speaker 118-1) maydynamically track the drift or the current time offset in thecoordination of the clocks in A/V hub 112 and speaker 118-1 (as well asthe negligible static offset associated with the propagation delay).

Once again, the current time offset may be determined by A/V hub 112 ormay be provided to A/V hub 112 by one or more of speakers 118 (such asspeaker 118-1). Note that in some embodiments the current time offset isfurther based on models of clock drift in A/V hub 112 and speakers 118.Then, A/V hub 112 may transmit, to speaker 118-1, one or more framesthat include audio content and playback timing information, which mayspecify playback times when speaker 118-1 is to playback the audiocontent based on the current time offset. This may be repeated for otherspeakers 118. Furthermore, the playback times of speakers 118 may have atemporal relationship so that the playback of the audio content byspeakers 118 is coordinated.

Moreover, note that the one or more additional communication techniquesmay also be used in these embodiments to identify and/or excludemulti-path wireless signals during the coordination of speakers 118.

As described further below with reference to FIGS. 8-10, anotherapproach for coordinating speakers 118 may use acoustic measurements. Inparticular, during a calibration mode, clocks in A/V hub 112 andspeakers 118 may be coordinated. Subsequently, A/V hub 112 may outputsound that corresponds to an acoustic-characterization pattern thatuniquely identifies A/V hub 112 (such as a sequence of pulses, differentfrequencies, etc.) at predefined transmission times. Thisacoustic-characterization pattern may be output at frequencies outsideof the range of human hearing (such as at ultrasonic frequencies).However, because of the relative drift in the clock in A/V hub 112, thesound corresponding to the acoustic-characterization pattern will bemeasured at speakers 118 (i.e., will arrive or be received) at differenttimes than the expected predefined transmission times based on theclocks in speakers 118. In these embodiments, the different times needto be corrected for the contributions associated with acousticpropagation delays based on the predetermined or known locations of A/Vhub 112 and speakers 118 and/or using wireless ranging. For example, thelocations may be determined using a triangulation and/or trilaterationin a local positioning system, a global positioning system, and/or awireless network (such as a cellular-telephone network or a WLAN). Thus,after correcting for the acoustic propagation delay, the difference ofthe arrival time and the predefined transmission time of a given frameat a given one of speakers 118 (such as speaker 118-1) may dynamicallytrack the drift or the current time offset in the coordination of theclocks in A/V hub 112 and speaker 118-1.

Alternatively or additionally, after the calibration mode, speakers 118may output sound that corresponds to acoustic-characterization patternsthat uniquely identify speakers 118 (such as different sequences ofpulses, different frequencies, etc.) at predefined transmission times.However, because of the relative drift in the clocks in speakers 118,the sound corresponding to the acoustic-characterization patterns willbe measured at A/V hub 112 (i.e., will arrive or be received) atdifferent times than the expected predefined transmission times based onthe clock in A/V hub 112. In these embodiments, the different times needto be corrected for the contributions associated with acousticpropagation delays based on the predetermined or known locations of A/Vhub 112 and speakers 118 and/or using wireless ranging. Thus, aftercorrecting for the acoustic propagation delay, the difference of thearrival time and the predefined transmission time of a given frame froma given one of speakers 118 (such as speaker 118-1) may dynamicallytrack the drift or the current time offset in the coordination of theclocks in A/V hub 112 and speaker 118-1.

Once again, the current time offset may be determined by A/V hub 112 ormay be provided to A/V hub 112 by speaker 118-1. Then, A/V hub 112 maytransmit, to speaker 118-1, one or more frames that include audiocontent and playback timing information, which may specify playbacktimes when speaker 118-1 is to playback the audio content based on thecurrent time offset. This may be repeated for other speakers 118.Furthermore, the playback times of speakers 118 may have a temporalrelationship so that the playback of the audio content by speakers 118is coordinated.

Although we describe the network environment shown in FIG. 1 as anexample, in alternative embodiments, different numbers or types ofelectronic devices may be present. For example, some embodiments includemore or fewer electronic devices. As another example, in anotherembodiment, different electronic devices are transmitting and/orreceiving packets or frames. While portable electronic device 110 andA/V hub 112 are illustrated with a single instance of radios 122, inother embodiments portable electronic device 110 and A/V hub 112 (andoptionally A/V display devices 114, receiver device 116, speakers 118and/or content sources 120) may include multiple radios.

We now describe embodiments of the communication technique. FIG. 2presents a flow diagram illustrating a method 200 for coordinatingplayback of audio content, which may be performed by an A/V hub, such asA/V hub 112 (FIG. 1). During operation, the A/V hub (such as a controlcircuit or control logic, e.g., a processor executing a program module,in the A/V hub) may receive, via wireless communication, frames(operation 210) or packets from one or more electronic devices, where agiven frame or packet includes a transmit time when a given electronicdevice transmitted the given frame or packet.

Then, the A/V hub may store receive times (operation 212) when theframes or packets were received, where the receive times are based on aclock in the A/V hub. For example, a receive time may be may be added toan instance of a packet or a frame or packet received from one of theelectronic devices by a physical layer and/or a media access control(MAC) layer in or associated with an interface circuit in the A/V hub.Note that the receive time may be associated with the leading edge orthe trailing edge of the packet or frame or packet, such as with areceive time signal which is associated with the leading edge or with areceive clear signal which is associated with the trailing edge.Similarly, the transmit time may be added to an instance of a frame or apacket transmitted by one of the electronic devices by a physical layerand/or a MAC layer in or associated with an interface circuit in theelectronic device. In some embodiments, the transmit and receive timesare determined and added to the frames or packets by wireless-rangingcapability in a physical layer and/or a MAC layer in or associated withthe interface circuits.

Moreover, the A/V hub may calculate current time offsets (operation 214)between clocks in the electronic devices and the clock in the A/V hubbased on the receive times and transmit times of the frames or packets.Furthermore, the current time offsets may be calculated by the A/V hubbased on models of clock drift in the electronic devices, such as anelectrical circuit model of a clock circuit and/or a look-up table ofclock drift as a function of time. Note that the electronic devices maybe located at non-zero distances from the A/V hub, and the current timeoffsets may be calculated based on the transmit times and the receivetimes using wireless ranging by ignoring the distances.

Next, the A/V hub may transmit one or more frames (operation 216) orpackets that include audio content and playback timing information tothe electronic devices, where the playback timing information specifiesplayback times when the electronic devices are to playback the audiocontent based on the current time offsets. Furthermore, the playbacktimes of the electronic devices may have a temporal relationship so thatthe playback of the audio content by the electronic devices iscoordinated. Note that the temporal relationship may have a non-zerovalue, so that at least some of the electronic devices are instructed toplayback the audio content with a phase relative to each other by usingdifferent values of the playback times. For example, the differentplayback times may be based on predetermined or dynamically determinedacoustic characteristics of an environment that includes the electronicdevices and the A/V hub. Alternatively or additionally, the differentplayback times may be based on a desired acoustic characteristic in theenvironment.

In some embodiments, the A/V hub optionally performs one or moreadditional operations (operation 218). For example, the electronicdevices may be located at vector distances from the A/V hub, and theinterface circuit may determine magnitudes of the vector distances basedon the transmit times and the receive times using wireless ranging.Moreover, the interface circuit may determine angles of the vectordistances based on the angle of arrival of wireless signals associatedwith the frames or packets that are received by the one or more antennasduring the wireless communication. Furthermore, the different playbacktimes may be based on the determined vector distances. For example, theplayback times may correspond to the determined vector distances suchthat the sound associated with the audio content from differentelectronic devices at different locations in the environment may arriveat a location in the environment (e.g., a location of the A/V hub, inthe middle of the environment, at a preferred listening location of auser, etc.) with a desired phase relationship or to achieve a desiredacoustic characteristic at the location.

Alternatively or additionally, the different playback times are based onan estimated location of a listener relative to the electronic devices,such that the sound associated with the audio content from differentelectronic devices at different locations in the environment may arriveat the estimated location of the listener with a desired phaserelationship or to achieve a desired acoustic characteristic at theestimated location. Techniques that can be used to determine thelocation of the listener are described further below with reference toFIGS. 14-16.

Note that while the wireless ranging capability in the interfacecircuits may involve coordinated clocks in the A/V hub and theelectronic devices, in other embodiments the clocks are not coordinated.Thus, a variety of radiolocation techniques may be used. In someembodiments, the wireless-ranging capability includes the use oftransmissions over GHz or multi-GHz bandwidths to create pulses of shortduration (such as, e.g., approximately 1 ns).

FIG. 3 is a drawing illustrating between A/V hub 112, and speaker 118-1.In particular, interface circuit 310 in speaker 118-1 may transmit oneor more frames or packets (such as packet 312) to A/V hub 112. Packet312 may include corresponding transmit time 314, based on an interfaceclock 316 provided by an interface clock circuit 318 in or associatedwith an interface circuit 310 in speaker 118-1, when speaker 118-1transmitted packets 312. When an interface circuit 320 in A/V hub 112receives packet 312, it may include receive time 322 in packet 312 (orit may store receive time 322 in memory 324), where for each packet thecorresponding receive time may be based on an interface clock 326provided by an interface clock circuit 328 in or associated withinterface circuit 318.

Then, interface circuit 320 may calculate, based on differences betweentransmit times 314 and receive times 322, a current time offset 330between interface clock 316 and interface clock 326. Moreover, interfacecircuit 320 may provide current time offset 330 to processor 332.(Alternatively, processor 332 may calculate the current time offset330.)

Furthermore, processor 332 may provide playback timing information 334and audio content 336 to interface circuit 320, where the playbacktiming information 334 specifies a playback time when speaker 118-1 isto playback audio content 336 based on the current time offset 330. Inresponse, interface circuit 330 may transmit one or more frames orpackets 338 that includes the playback timing information 334 and audiocontent 336 to speaker 118-1. (However, in some embodiments, playbacktiming information 334 and audio content 336 are transmitted usingseparate or different frames or packets.)

After interface circuit 310 receives the one or more frames or packets338, it may provide the playback timing information 334 and audiocontent 336 to processor 340. Processor 340 may execute software thatperforms a playback operation 342. For example, processor 340 may storeaudio content 336 in a queue in memory. In these embodiments, playbackoperation 350 includes outputting audio content 336 from the queue,including driving an electrical-to-acoustic transducer in speaker 118-1based on audio content 336 so speaker 118-1 outputs sound at a timespecified by the playback timing information 334.

In an exemplary embodiment, the communication technique is used tocoordinate the playback of audio content by speakers 118. This isillustrated in FIG. 4, which presents a drawing illustratingcoordinating playback of audio content by speakers 118. In particular,when frames or packets (such as packet 410-1) are transmitted byspeakers 118 they may include information specifying transmit times(such as transmit time 412-1). For example, the physical layer in theinterface circuits in speakers 118 may include the transmit times inpackets 410. In FIG. 4 and the other embodiments below, note thatinformation in frames or packets may be included at an arbitraryposition (such the beginning, the middle and/or the end).

When packets 410 are received by A/V hub 112, additional informationspecifying receive times (such as receive time 414-1 of packet 410-1)may be included in packets 410. For example, the physical layer in theinterface circuit in A/V hub 112 may include the receive times inpackets 410. Moreover, the transmit times and the receive times may beused to track the drift of clocks in A/V hub 112 and speakers 118.

Using the transmit times and receive times, A/V hub 112 may calculatecurrent time offsets between the clocks in speakers 118 and the clock inA/V hub 112. Furthermore, the current time offsets may be calculated byA/V hub 112 based on models in A/V hub 112 of the clock drift inspeakers 118. For example, a model of the relative or absolute clockdrift may include a polynomial or a cubic spline expression (and, moregenerally, a function) with parameters that specify or estimate theclock drift in a given speaker as a function of time based on historicaltime offsets.

Subsequently, A/V hub 112 may transmit one or more packets or frames orpackets that include audio content 420 and playback timing information(such as playback timing information 418-1 in packet 416-1) to speakers118, where the playback timing information specifies playback times whenspeakers 118 devices are to playback audio content 420 based on thecurrent time offsets. The playback times of speakers 118 may have atemporal relationship so that the playback of audio content 420 byspeakers 118 is coordinated, e.g., so that the associated sound orwavefronts arrive at a location 422 in an environment with a desiredphase relationship.

Another embodiment of the coordination in the communication technique isshown in FIG. 5, which presents a flow diagram illustrating a method 500for coordinating playback of audio content. Note that method 500 may beperformed by an A/V hub, such as A/V hub 112 (FIG. 1). During operation,the A/V hub (such as a control circuit or control logic, e.g., aprocessor executing a program module, in the A/V hub) may receive, viawireless communication, frames (operation 510) or packets fromelectronic devices.

Then, the A/V hub may store receive times (operation 512) when theframes or packets were received, where the receive times are based on aclock in the A/V hub. For example, a receive time may be may be added toan instance of a frame or a packet received from one of the electronicdevices by a physical layer and/or a MAC layer in or associated with aninterface circuit in the A/V hub. Note that the receive time may beassociated with the leading edge or the trailing edge of the frame or apacket, such as with a receive time signal which is associated with theleading edge or with a receive clear signal which is associated with thetrailing edge.

Moreover, the A/V hub may calculate current time offsets (operation 514)between clocks in the electronic devices and the clock in the A/V hubbased on the receive times and expected transmit times of the frames orpackets, where the expected transmit times are based on coordination ofthe clocks in the electronic devices and the clock in the A/V hub at aprevious time and a predefined transmit schedule of the frames orpackets (such as every 10 or 100 ms, which are nonlimiting examples).For example, during an initialization mode, time offsets between theclocks in the electronic devices and the clock in the A/V hub may beeliminated (i.e., coordination may be established). Note that thepredefined transmit times in the transmit schedule may include or may beother than beacon transmit times in a WLAN. Subsequently, the clocks andthe clock may have relative drift, which can be tracked based ondifferences between the receive times and expected transmit times of theframes or packets. In some embodiments, the current time offsets arecalculated by the A/V hub based on models of clock drift in theelectronic devices.

Next, the A/V hub may transmit one or more frames (operation 516) orpackets that include audio content and playback timing information tothe electronic devices, where the playback timing information specifiesplayback times when the electronic devices are to playback the audiocontent based on the current time offsets. Furthermore, the playbacktimes of the electronic devices may have a temporal relationship so thatthe playback of the audio content by the electronic devices iscoordinated. Note that the temporal relationship may have a non-zerovalue, so that at least some of the electronic devices are instructed toplayback the audio content with a phase relative to each other by usingdifferent values of the playback times. For example, the differentplayback times may be based on predetermined or dynamically determinedacoustic characteristics of an environment that includes the electronicdevices and the A/V hub. Alternatively or additionally, the differentplayback times may be based on a desired acoustic characteristic in theenvironment.

In some embodiments, the A/V hub optionally performs one or moreadditional operations (operation 518). For example, the electronicdevices may be located at vector distances from the A/V hub, and theinterface circuit may determine magnitudes of the vector distances basedon the transmit times and the receive times using wireless ranging.Moreover, the interface circuit may determine angles of the vectordistances based on the angle of arrival of wireless signals associatedwith the frames or packets that are received by the one or more antennasduring the wireless communication. Furthermore, the different playbacktimes may be based on the determined vector distances. For example, theplayback times may correspond to the determined vector distances suchthat the sound associated with the audio content from differentelectronic devices at different locations in the environment may arriveat a location in the environment (e.g., a location of the A/V hub, inthe middle of the environment, at a preferred listening location of auser, etc.) with a desired phase relationship or to achieve a desiredacoustic characteristic at the location.

Alternatively or additionally, the different playback times are based onan estimated location of a listener relative to the electronic devices,such that the sound associated with the audio content from differentelectronic devices at different locations in the environment may arriveat the estimated location of the listener with a desired phaserelationship or to achieve a desired acoustic characteristic at theestimated location. Techniques that can be used to determine thelocation of the listener are described further below with reference toFIGS. 14-16.

FIG. 6 is a drawing illustrating communication among portable electronicdevice 110, A/V hub 112, and speaker 118-1. In particular, during aninitialization mode, interface circuit 610 in A/V hub 112 may transmit aframe or packet 612 to interface circuit 614 in speaker 118-1. Thispacket may include information 608 that coordinates clocks 628 and 606provided, respectively, by interface clock circuits 616 and 618. Forexample, the information may eliminate a time offset between interfaceclock circuits 616 and 618 and/or may set interface clock circuits 616and 618 to the same clock frequency.

Subsequently, interface circuit 614 may transmit one or more frames orpackets (such as packet 620) to A/V hub 112 at predefined transmit times622.

When an interface circuit 610 in A/V hub 112 receives packet 620, it mayinclude receive time 624 in packet 620 (or it may store receive time 624in memory 626), where for each packet the corresponding receive time maybe based on interface clock 628 provided by an interface clock circuit616 in or associated with interface circuit 610.

Then, interface circuit 610 may calculate, based on differences betweentransmit times 622 and receive times 624, a current time offset 630between interface clock 628 and interface clock 606. Moreover, interfacecircuit 610 may provide current time offset 630 to processor 632.(Alternatively, processor 632 may calculate the current time offset630.)

Furthermore, processor 632 may provide playback timing information 634and audio content 636 to interface circuit 610, where the playbacktiming information 634 specifies a playback time when speaker 118-1 isto playback audio content 636 based on the current time offset 630. Inresponse, interface circuit 610 may transmit one or more frames orpackets 638 that includes the playback timing information 634 and audiocontent 636 to speaker 118-1. (However, in some embodiments, playbacktiming information 634 and audio content 636 are transmitted usingseparate or different frames or packets.)

After interface circuit 614 receives the one or more frames or packets638, it may provide the playback timing information 634 and audiocontent 636 to processor 640. Processor 640 may execute software thatperforms a playback operation 642. For example, processor 640 may storeaudio content 636 in a queue in memory. In these embodiments, playbackoperation 650 includes outputting audio content 636 from the queue,including driving an electrical-to-acoustic transducer in speaker 118-1based on audio content 636 so speaker 118-1 outputs sound at a timespecified by the playback timing information 634.

In an exemplary embodiment, the communication technique is used tocoordinate the playback of audio content by speakers 118. This isillustrated in FIG. 7, which presents a drawing illustratingcoordinating playback of audio content by speakers 118. In particular,A/V hub 112 may transmit frames or packets 710 to speakers 118 withinformation (such as information 708 in packet 710-1) that coordinatesclocks, provided by clock circuits, in A/V hub 112 and speakers 118.

Subsequently, speakers 118 may transmit frames or packets 712 to A/V hub112 at predefined transmit times. When these frames or packets arereceived by A/V hub 112, information specifying receive times may beincluded in packets 712 (such as receive time 714-1 in packet 712-1).The predefined transmit times and the receive times may be used to trackthe drift of the clocks in A/V hub 112 and speakers 118.

Using the predefined transmit times and the receive times, A/V hub 112may calculate current time offsets between the clocks in speakers 118and the clock in A/V hub 112. Furthermore, the current time offsets maybe calculated by A/V hub 112 based on models in A/V hub 112 of the clockdrift in speakers 118. For example, a model of the relative or absoluteclock drift may include a polynomial or a cubic spline expression (and,more generally, a function) with parameters that specify or estimate theclock drift in a given speaker as a function of time based on historicaltime offsets.

Subsequently, A/V hub 112 may transmit one or more frames or packetsthat include audio content 720 and playback timing information (such asplayback timing information 718-1 in packet 716-1) to speakers 118,where the playback timing information specifies playback times whenspeakers 118 devices are to playback audio content 720 based on thecurrent time offsets. The playback times of speakers 118 may have atemporal relationship so that the playback of audio content 720 byspeakers 118 is coordinated, e.g., so that the associated sound orwavefronts arrive at a location 722 in an environment with a desiredphase relationship.

Another embodiment of the coordination in the communication technique isshown in FIG. 8, which presents a flow diagram illustrating a method 800for coordinating playback of audio content. Note that method 800 may beperformed by an A/V hub, such as A/V hub 112 (FIG. 1). During operation,the A/V hub (such as a control circuit or control logic, e.g., aprocessor executing a program module, in the A/V hub) may measure sound(operation 810) output by electronic devices in an environment thatincludes the A/V hub using one or more acoustic transducers in the A/Vhub, where the sound corresponds to one or moreacoustic-characterization patterns. For example, the measured sound mayinclude the sound pressure. Note that the acoustic-characterizationpatterns may include pulses. Moreover, the sound may be in a range offrequencies outside of human hearing, such as ultrasound.

Furthermore, a given electronic device may output the sound at adifferent time in the one or more times than those used by a remainderof the electronic devices, so that the sound from the given electronicdevice can be identified or distinguished from the sound output by theremainder of the electronic devices. Alternatively or additionally, thesound output by a given electronic device may correspond to a givenacoustic-characterization pattern, which may be different from thoseused by the remainder of the electronic devices. Thus, theacoustic-characterization patterns may uniquely identify the electronicdevices.

Then, the A/V hub may calculate current time offsets (operation 812)between clocks in the electronic devices and a clock in the A/V hubbased on the measured sound, one or more times when the electronicdevices output the sound and the one or more acoustic-characterizationpatterns. For example, the A/V hub may correct the measured sound basedon an acoustic characteristic of the environment, such as an acousticdelay associated with at least a particular frequency or a predetermined(or dynamically determined) acoustic transfer function of theenvironment in at least a band of frequencies (such as 100-20,000 Hz,which is a nonlimiting example), and the output times may be compared totriggered output times or predefined output times. This may allow theA/V hub to determine the original output sound without the spectralfiltering or distortions associated with the environment, which mayallow the A/V hub to more accurately determine the current time offsets.

Note that the measured sound may include information that specifies theone or more times when the electronic devices output the sound (e.g.,the pulses in the acoustic-characterization patterns may specify thetimes), and the one or more times may correspond to the clocks in theelectronic devices. Alternatively or additionally, the A/V hub mayoptionally provide to the electronic devices, via the wirelesscommunication, one or more times (operation 808) when the electronicdevices are to output the sound, and the one or more times maycorrespond to the clock in the A/V hub. For example, the A/V hub maytransmit one or more frames or packets to the electronic devices withthe one or more times. Thus, the A/V hub may trigger the output of thesound or the sound may be output at predefined output times.

Next, the A/V hub may transmit, using wireless communication, one ormore frames (operation 814) or packets that include audio content andplayback timing information to the electronic devices, where theplayback timing information specifies playback times when the electronicdevices are to playback the audio content based on the current timeoffsets. Moreover, the playback times of the electronic devices have atemporal relationship so that the playback of the audio content by theelectronic devices is coordinated. Note that the temporal relationshipmay have a non-zero value, so that at least some of the electronicdevices are instructed to playback the audio content with a phaserelative to each other by using different values of the playback times.For example, the different playback times may be based on predeterminedor dynamically determined acoustic characteristics of an environmentthat includes the electronic devices and the A/V hub. Alternatively oradditionally, the different playback times may be based on a desiredacoustic characteristic in the environment and/or an estimated locationof a listener relative to the electronic devices.

In some embodiments, the A/V hub optionally performs one or moreadditional operations (operation 816). For example, the A/V hub maymodify the measured sound based on an acoustic transfer function of theenvironment in at least a band of frequencies that includes the spectralcontent in acoustic-characterization patterns. Note that the acoustictransfer function may be predetermined and accessed by the A/V hub ordynamically determined by the A/V hub. This correction for the filteringassociated with the environment may be necessary because, while the timedelay and dispersion associated with the propagation of sound in theenvironment may be much larger than the desired coordination of theclocks in the electronic devices and the clock in the A/V hub, theleading edge of the modified direct sound may be determined withsufficient accuracy that the current time offset between the clocks inthe electronic devices and the clock in the A/V hub can be determined.For example, the desired coordination accuracy of speakers 118 may be assmall as on the order of a microsecond, while the propagation delay ofsound in a typical room (e.g., over distances of at most 10-30 m) may befive orders of magnitude larger. Nonetheless, the modified measuredsound may allow the leading edges of the direct sound associated withpulses in the sound output from a given electronic device to be measuredwith as little as microsecond accuracy, which can facilitatecoordination of the clocks in the electronic devices and the clock inthe A/V hub. In some embodiments, the A/V hub determines the temperaturein the environment, and the calculations of the current time offset maybe corrected for changes in the temperature (which impact the speed ofsound in the environment).

FIG. 9 is a drawing illustrating communication among portable electronicdevice 110, A/V hub 112, and speaker 118-1. In particular, processor 910in speaker 118-1 may instruct 912 one or more acoustic transducers 914in speaker 118-1 to output sound at an output time, where the soundcorresponds to an acoustic-characterization pattern. For example, theoutput time may be predefined (such as based on a pattern or sequence ofpulses in the acoustic-characterization pattern, a predefined outputschedule with scheduled output times or a predefined interval betweenoutput times) and thus may be known to A/V hub 112 and the speaker118-1. Alternatively, interface circuit 916 in A/V hub 112 may provide atrigger frame or packet 918. After interface circuit 920 receivestrigger packet 918, it may forward an instruction 922 to processor 910in speaker 118-1, which triggers the sound output from the one or moreacoustic transducers 914 based on instruction 922.

Subsequently, the one or more acoustic transducers 924 in A/V hub 112may measure 926 the sound, and may provide information 928 thatspecifies the measurements to processor 930 in A/V hub 112.

Next, processor 930 may calculate a current time offset 932 between aclock from a clock circuit in speaker 118-1 (such as an interface clockcircuit) and a clock from a clock circuit (such as an interface clockcircuit) in A/V hub 112 based on the information 928, one or more timeswhen speaker 118-1 output the sound and an acoustic-characterizationpattern associated with speaker 118-1. For example, processor 930 maydetermine the current time offset 932 based on at least two times in theacoustic-characterization pattern when the one or more acoustictransducers 914 in speaker 118-1 output sound corresponding to theacoustic-characterization pattern.

Moreover, processor 930 may provide playback timing information 934 andaudio content 936 to interface circuit 916, where the playback timinginformation 934 specifies a playback time when speaker 118-1 is toplayback audio content 936 based on the current time offset 932. Notethat processor 930 may access audio content 936 in memory 938. Inresponse, interface circuit 916 may transmit one or more frames orpackets 940 that includes the playback timing information 934 and audiocontent 936 to speaker 118-1. (However, in some embodiments, playbacktiming information 934 and audio content 936 are transmitted usingseparate or different frames or packets.)

After interface circuit 920 receives the one or more frames or packets940, it may provide the playback timing information 934 and audiocontent 936 to processor 924. Processor 924 may execute software thatperforms a playback operation 942. For example, processor 924 may storeaudio content 936 in a queue in memory. In these embodiments, playbackoperation 942 includes outputting audio content 936 from the queue,including driving one or more of acoustic transducers 914 based on audiocontent 936 so speaker 118-1 outputs sound at a time specified by theplayback timing information 934.

In an exemplary embodiment, the communication technique is used tocoordinate the playback of audio content by speakers 118. This isillustrated in FIG. 10, which presents a drawing illustratingcoordinating playback of audio content by speakers 118. In particular,speakers 118 may output sound 1010 corresponding toacoustic-characterization patterns. For example, anacoustic-characterization pattern associated with speaker 118-1 mayinclude two or more pulses 1012, where a time interval 1014 betweenpulses 1012 may correspond to a clock provided by a clock circuit inspeaker 118-1. In some embodiments, a pattern or sequence of pulses inthe acoustic-characterization patterns may also uniquely identifyspeakers 118. While pulses 1012 are used to illustrated theacoustic-characterization patterns in FIG. 10, in other embodiments avariety of temporal, frequency and/or modulation techniques may be used,including: amplitude modulation, frequency modulation, phase modulation,etc. Note that A/V hub 112 may optional trigger the output of sound 1010by transmitting one or more frames or packets 1016 with information 1018specifying times to speakers 118 when speakers 118 are to output sound1010 corresponding to the acoustic-characterization patterns.

Then, A/V hub 112 may measure sound 1010 output by the electronicdevices using one or more acoustic transducers, where the soundcorresponds to one or more of the acoustic-characterization patterns.After measuring sound 1010, A/V hub 112 may calculate current timeoffsets between clocks in speakers 118 and a clock in A/V hub 112 basedon the measured sound 1010, one or more times when the speakers 118output the sound and the one or more acoustic-characterization patterns.In some embodiments, the current time offsets may be calculated by A/Vhub 112 based on models in A/V hub 112 of clock drift in speakers 118.For example, a model of the relative or absolute clock drift may includea polynomial or a cubic spline expression (and, more generally, afunction) with parameters that specify or estimate the clock drift in agiven speaker as a function of time based on historical time offsets.

Next, A/V hub 112 may transmit one or more frames or packets thatinclude audio content 1022 and playback timing information to speakers118 (such as playback timing information 1024-1 in packet 1020-1), wherethe playback timing information specifies playback times when speakers118 devices are to playback audio content 1022 based on the current timeoffsets. The playback times of speakers 118 may have a temporalrelationship so that the playback of audio content 1022 by speakers 118is coordinated, e.g., so that the associated sound or wavefronts arriveat a location 1026 in an environment with a desired phase relationship.

The communication technique may include operations that are used toadapt the coordination to improve the acoustic experience of listeners.One approach is shown in FIG. 11, which presents a flow diagramillustrating a method 1100 for selectively determining one or moreacoustic characteristics of an environment (such as a room). Method 1100may be performed by an A/V hub, such as A/V hub 112 (FIG. 1). Duringoperation, the A/V hub (such as a control circuit or control logic,e.g., a processor executing a program module, in the A/V hub) mayoptionally detect, using wireless communication, an electronic device(operation 1110) in an environment. Alternatively or additionally, theA/V hub may determine a change condition (operation 1112), where thechange condition includes: that the electronic device was not previouslydetected in the environment; and/or a change in a location of theelectronic device (including a change in the location that occurs longafter the electronic device was first detected in the environment).

When the change condition is determined (operation 1112), the A/V hubmay transition into a characterization mode (operation 1114). During thecharacterization mode, the A/V hub may: provide instructions (operation1116) to the electronic device to playback audio content at a specifiedplayback time; determine one or more acoustic characteristics (operation1118) of the environment based on acoustic measurements in theenvironment; and store the characterization information (operation 1120)in memory, where the characterization information includes the one ormore acoustic characteristics.

Moreover, the A/V hub may transmit one or more frames (operation 1122)or packets that include additional audio content and playback timinginformation to the electronic device, where the playback timinginformation may specify a playback time when the electronic device is toplayback the additional audio content based on the one or more acousticcharacteristics.

In some embodiments, the A/V hub optionally performs one or moreadditional operations (operation 1124). For example, the A/V hub maycalculate the location of the electronic device in the environment, suchas based on wireless communication. Moreover, the characterizationinformation may include an identifier of the electronic device, whichmay be received from the electronic device by the A/V hub using wirelesscommunication.

Furthermore, the A/V hub may determine the one or more acousticcharacteristics based, at least in part, on acoustic measurementsperformed by other electronic devices. Thus, the A/V hub may communicatewith the other electronic devices in the environment using the wirelesscommunication, and may receive the acoustic measurements from the otherelectronic devices. In these embodiments, the one or more acousticcharacteristics may be determined based on locations of the otherelectronic devices in the environment. Note that the A/V hub may:receive the locations of the other electronic devices from the otherelectronic devices; access predetermined locations of the otherelectronic devices stored in memory; and determine the locations of theother electronic devices, e.g., based on the wireless communication.

In some embodiments, the A/V hub includes one or more acoustictransducers, and the A/V hub performs the acoustic measurements usingthe one or more acoustic transducers. Therefore, the one or moreacoustic characteristics may be determined by the A/V hub alone or inconjunction with the acoustic measurements performed by the otherelectronic devices.

However, in some embodiments, instead of determining the one or moreacoustic characteristics, the A/V hub receives the determined one ormore acoustic characteristics from one of the other electronic devices.

While the acoustic characterization may be fully automated based on thechange condition, in some embodiments a user may manually initiate thecharacterization mode or may manually approve the characterization modewhen the change condition is detected. For example, the A/V hub may:receive a user input; and transition into the characterization modebased on the user input.

FIG. 12 is a drawing illustrating communication between A/V hub 112 andspeaker 118-1. In particular, interface circuit 1210 in A/V hub 112 maydetect speaker 118-1 by wireless communication of frames or packets 1212with interface circuit 1214 in speaker 118-1. Note that thiscommunication may be unilateral or bilateral.

Interface circuit 1210 may provide information 1216 to processor 1218.This information may indicate the presence of speaker 118-1 in anenvironment. Alternatively or additionally, information 1216 may specifya location of speaker 118-1.

Then, processor 1218 may determine whether a change condition 1220 hasoccurred. For example, processor 1218 may determine the presence ofspeaker 118-1 in the environment when it was not present previously orthat the location of previously detected speaker 118-1 has changed.

When change condition 1220 is determined, processor 1218 may transitionto a characterization mode 1222. During characterization mode 1222,processor 1218 may provide instruction 1224 to interface circuit 1210.In response, interface circuit 1210 may transmit instruction 1224 tointerface circuit 1214 in frame or packet 1226.

After receiving packet 1226, interface circuit 1214 may provideinstruction 1224 to processor 1228, when then instructs one or moreacoustic transducers 1230 to playback audio content 1232 at a specifiedplayback time. Note that processor 1228 may access audio content 1232 inmemory 1208 or audio content 1232 may be included in packet 1226. Next,one or more acoustic transducers 1234 in A/V hub 112 may performacoustic measurements 1236 of sound corresponding to audio content 1232output by the one or more acoustic transducers 1230. Based on acousticmeasurements 1236 (and/or additional acoustic measurements received fromother speakers by interface circuit 1210), processor 1218 may determineone or more acoustic characteristics 1238 of the environment, which arethen stored in memory 1240.

Moreover, processor 1218 may provide playback timing information 1242and audio content 1244 to interface circuit 1210, where the playbacktiming information 1242 specifies a playback time when speaker 118-1 isto playback audio content 1244 based, at least in part, on the one ormore acoustic characteristics 1238. In response, interface circuit 1210may transmit one or more frames or packets 1246 that includes theplayback timing information 1242 and audio content 1244 to speaker118-1. (However, in some embodiments, playback timing information 1242and audio content 1244 are transmitted using separate or differentframes or packets.)

After interface circuit 1214 receives the one or more frames or packets1246, it may provide the playback timing information 1242 and audiocontent 1244 to processor 1228. Processor 1228 may execute software thatperforms a playback operation 1248. For example, processor 1228 maystore audio content 1244 in a queue in memory. In these embodiments,playback operation 1248 includes outputting audio content 1244 from thequeue, including driving one or more of acoustic transducers 1230 basedon audio content 1244 so speaker 118-1 outputs sound at a time specifiedby the playback timing information 1242.

In an exemplary embodiment, the communication technique is used toselectively determine one or more acoustic characteristics of anenvironment (such as a room) that includes A/V hub 112 when a change isdetected. FIG. 13 presents a drawing illustrating selective acousticcharacterization of an environment that includes speakers 118. Inparticular, A/V hub 112 may detect speaker 118-1 in the environment. Forexample, A/V hub 112 may detect speaker 118-1 based on wirelesscommunication of one or more frames or packets 1310 with speaker 118-1.Note that the wireless communication may be unilateral or bilateral.

When a change condition is determined (such as when the presence ofspeaker 118-1 is first detected, i.e., when speaker 118-1 was notpreviously detected in the environment, and/or when there is a change ina location 1312 of previously detected speaker 118-1 in theenvironment), A/V hub 112 may transition into a characterization mode.For example, A/V hub 112 may transition into the characterization modewhen a magnitude change in location 1312 on the order of the wavelengthat the upper limit of human hearing, e.g., a change of 0.0085, 0.017 or0.305 m (which are nonlimiting examples), in location 1312 of speaker118-1 is detected.

During the characterization mode, A/V hub 112 may: provide instructionsin frame or packet 1314 to speaker 118-1 to playback audio content at aspecified playback time (i.e., to output sound 1316); determine one ormore acoustic characteristics of the environment based on acousticmeasurements of sound 1316 output by speaker 118-1; and store the one ormore acoustic characteristics, which may include location 1312 ofspeaker 118-1, in memory.

For example, the audio content may include a pseudorandom frequencypattern or white noise over a range of frequencies (such as between 100and 10,000 or 20,000 Hz, or two or more sub-frequency bands in the rangeof human hearing, e.g., at 500, 1000 and 2000 Hz, which are nonlimitingexamples), an acoustic pattern having a carrier frequency that varies asa function of time over a range of frequencies, an acoustic patternhaving spectral content in a range of frequencies, and/or one or moretypes of music (such as symphony music, classical music, chamber music,opera, rock or pop music, etc.). In some embodiments, the audio contentuniquely identifies speaker 118-1, such as a particular temporalpattern, spectral content and/or one or more frequency tones.Alternatively or additionally, A/V hub 112 may receive, via wirelesscommunication with speaker 118-1, an identifier of speaker 118-1, suchas an alphanumeric code.

However, in some embodiments, the acoustic characterization is performedwithout speaker 118-1 playing the audio content. For example, theacoustic characterization may be based on the acoustic energy associatedwith a person's voice or by measuring 1-2 min. of percussive backgroundnoise in the environment. Thus, in some embodiments the acousticcharacterization includes passive characterization (instead of activemeasurements when the audio content is playing).

Moreover, the acoustic characteristics may include: an acoustic spectralresponse of the environment over a range of frequencies (i.e.,information that specifies an amplitude response as a function offrequency), an acoustic transfer function or impulse response over arange of frequencies (i.e., information that specifies an amplitude anda phase response as a function of frequency), room resonances orlow-frequency room modes (which have nodes and antinodes as a functionof position or location in the environment, and which may be determinedby measuring sound in the environment in different directions at 90°from each other), location 1312 of speaker 118-1, reflections (includingearly reflections within 50-60 ms of the arrival of direct sound fromspeaker 118-1, and late reflections or echoes that occur on longer timescales, which can impact clarity), an acoustic delay of the directsound, an average reverberation time over a range of frequencies (or thepersistence of acoustic sound in the environment over a range offrequencies after the audio content has discontinued), a volume of theenvironment (such as a size and/or a geometry of room, which may bedetermined optically), background noise in the environment, ambientsound in the environment, a temperature of the environment, a number ofpeople in the environment (and, more generally, absorption or acousticloss over a range of frequencies in the environment), a metric of howacoustically lively, bright or dull the environment is and/orinformation that specifies a type of the environment (such as anauditorium, a general-purpose room, a concert hall, a size of a room,types of furnishing in a room, etc.). For example, the reverberationtime may be defined as the time for the sound pressure associated withan impulse at a frequency to decay to a particular level, such as −60dB. In some embodiments, the reverberation time is a function of thefrequency. Note that the range of frequencies in the preceding examplesof the acoustic characteristics may be the same or different from eachother. Thus, in some embodiments, different ranges of frequencies may beused for different acoustic characteristics. In addition, note that an‘acoustic transfer function’ in some embodiments may include a magnitudeof the acoustic transfer function (which is sometimes referred to as an‘acoustic spectral response’), a phase of the acoustic transferfunction, or both.

As noted previously, the acoustic characteristics may include location1312 of speaker 118-1. The location 1312 of speaker 118-1 (includingdistance and direction) may be determined by A/V hub 112 and/or inconjunction with other electronic devices in the environment (such asspeakers 118) using techniques such as: triangulation, trilateration,time of flight, wireless ranging, the angle of arrival, etc. Moreover,location 1312 may be determined by A/V hub 112 using: wirelesscommunication (such as communication with a wireless local area networkor with a cellular-telephone network), acoustic measurements, a localpositioning system, a global positioning system, etc.

While the acoustic characteristics may be determined by A/V hub 112based on measurements performed by A/V hub 112, in some embodiments theacoustic characteristics are determined by or in conjunction with otherelectronic devices in the environment. In particular, one or more otherelectronic devices (such as one or more other speakers 118) may performacoustic measurements, which are then wirelessly communicated to A/V hub112 in frames or packets 1318. (Thus, acoustic transducers that performthe acoustic measurements may be included in A/V hub 112 and/or in theone or more other speakers 118.) Consequently, A/V hub 112 may computethe acoustic characteristics based, at least in part, on the acousticmeasurements performed by A/V hub 112 and/or the one or more otherspeakers 118. Note that the computations may also be based onlocation(s) 1320 of the one or more other speakers 118 in theenvironment. These locations may be: received from the one or more otherspeakers 118 in frames or packets 1318, calculated using one of theaforementioned techniques (such as using wireless ranging), and/oraccessed in memory (i.e., locations 1320 may be predetermined).

Moreover, while the acoustic characterization may occur when the changecondition is detected, alternatively or additionally A/V hub 112 maytransition to the characterization mode based on a user input. Forexample, the user may activate a virtual command icon in a userinterface on portable electronic device 110. Thus, the acousticcharacterization may be automatically, manually initiated and/orsemi-automatically initiated (in which a user interface is used to askthe user for approval before the transition to the characterizationmode).

After determining the acoustic characteristics, A/V hub 112 maytransition back to a normal operating mode. In this operating mode, A/Vhub 112 may transmit one or more frames or packets (such as packet 1322)that include additional audio content 1324 (such as music) and playbacktiming information 1326 to speaker 118-1, where the playback timinginformation 1326 may specify a playback time when speaker 118-1 is toplayback the additional audio content 1324 based on the one or moreacoustic characteristics. Thus, the acoustic characterization may beused to correct for or adapt to the changes (direct or indirect) in theone or more acoustic characteristics that are associated with a changein location 1312 of speaker 118-1, thereby improving the userexperience.

Another approach for improving the acoustic experience is to adapt thecoordination based on dynamically tracked locations of one or morelisteners. This is shown in FIG. 14, which presents a flow diagramillustrating a method 1400 for calculating an estimated location. Notethat method 1400 may be performed by an A/V hub, such as A/V hub 112(FIG. 1). During operation, the A/V hub (such as a control circuit orcontrol logic, e.g., a processor executing a program module, in the A/Vhub) may calculate an estimated location of a listener (operation 1410)(or an electronic device associated with the listener, such as portableelectronic device 110 in FIG. 1) relative to the electronic devices inan environment that includes the A/V hub and the electronic devices.

Then, the A/V hub may transmit one or more frames (operation 1412) orpackets that include audio content and playback timing information tothe electronic devices, where the playback timing information specifiesplayback times when the electronic devices are to playback the audiocontent based on the estimated location. Furthermore, the playback timesof the electronic devices have a temporal relationship so that theplayback of the audio content by the electronic devices is coordinated.Note that the temporal relationship may have a non-zero value, so thatat least some of the electronic devices are instructed to playback theaudio content with a phase relative to each other by using differentvalues of the playback times. For example, the different playback timesmay be based on predetermined or dynamically determined acousticcharacteristics of the environment that includes the electronic devicesand the A/V hub. Alternatively or additionally, the different playbacktimes may be based on a desired acoustic characteristic in theenvironment. Additionally, the playback times may be based on currenttime offsets between clocks in the electronic devices and a clock in theA/V hub.

In some embodiments, the A/V hub optionally performs one or moreadditional operations (operation 1414). For example, the A/V hub maycommunicate with another electronic device, and the estimated locationof the listener may be calculated based on the communication with theother electronic device.

Moreover, the A/V hub may include an acoustic transducer that performssound measurements in the environment, and the estimated location of thelistener may be calculated based on the sound measurements.Alternatively or additionally, the A/V hub may communicate with otherelectronic devices in the environment and may receive additional soundmeasurements of the environment from the other electronic devices, andthe estimated location of the listener may be calculated based on theadditional sound measurements.

In some embodiments, the A/V hub performs time-of-flight measurements,and the estimated location of the listener is calculated based on thetime-of-flight measurements.

Furthermore, the A/V hub may calculate additional estimated locations ofadditional listeners relative to the electronic devices in theenvironment, and the playback times may be based on the estimatedlocation and the additional estimated locations. For example, theplayback times may be based on an average of the estimated location andthe additional estimated locations. Alternatively, the playback timesmay be based on a weighted average of the estimated location and theadditional estimated locations.

FIG. 15 is a drawing illustrating communication among portableelectronic device 110, A/V hub 112, and speakers 118, such as speaker118-1. In particular, interface circuit 1510 in A/V hub 112 may receiveone or more frames or packets 1512 from interface circuit 1514 inportable electronic device 110. Note that the communication between A/Vhub 112 and portable electronic device 110 may be unidirectional orbidirectional. Then, based on the one or more frames or packets 1512,interface circuit 1510 and/or processor 1516 in A/V hub 112 may estimatelocation 1518 of a listener associated with portable electronic device110. For example, interface circuit 1510 may provide information 1508based on packets 1512, which is used by processor 1516 to estimatelocation 1518.

Alternatively or additionally, one or more acoustic transducers 1520 inA/V hub 112 and/or one or more acoustic transducers 1506 in speakers 118may performs measures 1522 of sound associated with listener. Ifspeakers 118 perform measurements 1522-2 of the sound, interfacecircuits 1524 in one or more of speakers 118 (such as speaker 118-1) maytransmit one or more frames or packets 1526 to interface circuit 1510with information 1528 that specifies measurements 1522-2 of the soundbased on instructions 1530 from processor 1532. Then, interface circuit1514 and/or processor 1516 may estimate location 1518 based on themeasured sound 1522.

Next, processor 1516 may instruct interface circuit 1510 to transmit oneor more frames or packets 1536 to speaker 118-1 with playback timinginformation 1538 and audio content 1540, where the playback timinginformation 1538 specifies a playback time when speaker 118-1 is toplayback audio content 1540 based, at least in part, on location 1518.(However, in some embodiments, playback timing information 1538 andaudio content 1540 are transmitted using separate or different frames orpackets.) Note that processor 1516 may access audio content 1540 inmemory 1534.

After receiving the one or more frames or packets 1536, interfacecircuit 1524 may provide playback timing information 1538 and audiocontent 1540 to processor 1532. Processor 1532 may execute software thatperforms a playback operation 1542. For example, processor 1532 maystore audio content 1540 in a queue in memory. In these embodiments,playback operation 1542 includes outputting audio content 1540 from thequeue, including driving one or more of acoustic transducers 1506 basedon audio content 1540 so speaker 118-1 outputs sound at a time specifiedby the playback timing information 1538.

In an exemplary embodiment, the communication technique is used todynamically track the locations of one or more listeners in anenvironment. FIG. 16 presents a drawing illustrating calculating anestimated location of one or more listeners relative to speakers 118. Inparticular, A/V hub 112 may calculate estimated location(s) of one ormore listeners, such as location 1610 of listener 1612 relative to suchas speakers 118 in an environment that includes A/V hub 112 and speakers118. For example, location 1610 may be determined coarsely (e.g., to thenearest room, 3-10 m accuracy, etc.) or finely (e.g., 0.1-3 m accuracy),which are nonlimiting numerical examples.

In general, location 1610 may be determined by A/V hub 112 and/or inconjunction with other electronic devices (such as speakers 118) in theenvironment using techniques such as: triangulation, trilateration, timeof flight, wireless ranging, the angle of arrival, etc. Moreover,location 1610 may be determined by A/V hub 112 using: wirelesscommunication (such as communication with a wireless local area networkor with a cellular-telephone network), acoustic measurements, a localpositioning system, a global positioning system, etc.

For example, location 1610 of at least listener 1612 may be estimated byA/V hub 112 based on wireless communication (such as using wirelessranging, time-of-flight measurements, the angle of arrival, RSSI, etc.)of one or more frames or packets 1614 with another electronic device,such as portable electronic device 110, which may be proximate tolistener 1612 or on their person. In some embodiments, the wirelesscommunication with the other electronic device (such as a MAC address inframes or packets received from portable electronic device 110) is usedas a signature or an electronic thumbprint that identifies listener1612. Note that the communication between portable electronic device 110and A/V hub 112 may be unidirectional or bidirectional.

During wireless ranging, A/V hub 112 may transmit a frame or a packetthat includes a transmission time to, e.g., portable electronic device110. When this frame or packet is received by portable electronic device110, the arrival time may be determined. Based on the product of thetime of flight (the difference of the arrival time and the transmissiontime) and the speed of propagation, the distance between A/V hub 112 andportable electronic device 110 can be calculated. Then, this distancemay be communicated in a subsequent transmission of a frame or a packetfrom portable electronic device 110 to A/V hub 112 along with anidentifier of portable electronic device 110. Alternatively, portableelectronic device 110 may transmit a frame or a packet that includes atransmission time and an identifier of portable electronic device 110,and A/V hub 112 may determine the distance between portable electronicdevice 110 and A/V hub 112 based on the product of the time of flight(the difference of a arrival time and the transmission time) and thespeed of propagation.

In a variation on this approach, A/V hub 112 may transmit frames orpackets 1614 that are reflected at portable electronic device 110, andthe reflected frames or packets 1614 may be used to dynamicallydetermine the distance between portable electronic device 110 and A/Vhub 112.

While the preceding example illustrated wireless ranging withcoordinated clocks in portable electronic device 110 and A/V hub 112, inother embodiments the clocks are not coordinated. For example, theposition of portable electronic device 110 may be estimated based on thespeed of propagation and the time of arrival data of wireless signals atseveral receivers at different known locations in the environment (whichis sometimes referred to as ‘differential time of arrival’) even whenthe transmission time is unknown or unavailable. For example, thereceivers may be at least some of the other speakers 118 at locations1616, which may be predefined or predetermined. More generally, avariety of radiolocation techniques may be used, such as: determiningdistance based on a difference in the power of the RSSI relative to theoriginal transmitted signal strength (which may include corrections forabsorption, refraction, shadowing and/or reflection); determining theangle of arrival at a receiver (including non-line-of-sight reception)using a directional antenna or based on the differential time of arrivalat an array of antennas with known location(s) in the environment;determining the distance based on backscattered wireless signals; and/ordetermining the angle of arrival at two receivers having known locationin the environment (i.e., trilateration or multilateration). Note thatthe wireless signals may include transmissions over GHz or multi-GHzbandwidths to create pulses of short duration (such as, e.g.,approximately 1 ns), which may allow the distance to be determinedwithin 0.305 m (e.g., 1 ft), and which are nonlimiting examples. In someembodiments, the wireless ranging is facilitated using locationinformation, such as a location of one or more of electronic devices inthe environment (such as locations 1616) that are determined orspecified by a local positioning system, a global positioning systemand/or a wireless network.

Alternatively or additionally, location 1610 may be estimated by A/V hub112 based on sound measurements in the environment, such as acoustictracking of listener 1612, e.g., based on sounds 1618 they make as theymove about, talk and/or breathe. The sound measurements may be performedby A/V hub 112 (such as using two or more acoustic transducers, e.g.,microphones, which may be arranged as a phased array). However, in someembodiments sound measurements may be performed separately oradditionally by one or more electronic devices in the environment, suchas speakers 118, and these sound measurements may be wirelesscommunicated to A/V hub 112 in frames or packets 1618, which then usesthe sound measurements to estimate location 1610. In some embodiments,listener 1612 is identified using a voice-recognition technique.

In some embodiments, location 1610 is estimated by A/V hub 112 based onsound measurements in the environment and a predetermined acousticcharacteristic of the environment, such as a spectral response or anacoustic transfer function. For example, variation in the excitation ofpredetermined room modes as listener 1612 moves in the environment maybe used to estimate location 1610.

Moreover, one or more other techniques may be used to track or estimatelocation 1610 of listener 1612. For example, location 1610 may beestimated based on optical imaging of listener 1612 in a band ofwavelengths (such as visible light or infrared light), time-of-flightmeasurements (such as laser ranging), and/or a grid of optical beams(such as infrared beams) that localize listener 1612 in a grid (and,thus, coarsely determine location 1610) based on a pattern of beam-linecrossings. In some embodiments, the identity of listener 1612 isdetermined in optical images using a facial-recognition and/or agate-recognition technique.

For example, in some embodiments the location of the listener in theenvironment is tracked based on wireless communication with a cellulartelephone that is carried with the listener. Based on the pattern of thelocations in the environment, the locations of furniture in theenvironment and/or a geometry of the environment (such as a size ordimensions of a room) may be determined. This information may be used todetermine an acoustic characteristic of the environment. Moreover, thehistorical locations of the listener may be used to constrain anestimated location of the listener in the environment. In particular,historical information about the location of the listener in theenvironment at different times of day may be used to assist inestimating the current location of the listener at a particular time ofday. Thus, in general, the location of the listener may be estimatedusing a combination of optical measurements, acoustic measurements,acoustic characteristics, wireless communication and/or machinelearning.

After determining location 1610, A/V hub 112 may transmit at least oneor more frames or packets to speakers 118 that include additional audiocontent 1622 (such as music) and playback timing information (such asplayback timing information 1624-1 in packet 1620-1 to speaker 118-1),where the playback timing information 1624-1 may specify a playback timewhen speaker 118-1 is to playback the additional audio content 1622based on location 1610. Thus, the communication technique may be used tocorrect for or adapt to the changes in location 1610, thereby improvingthe user experience.

As noted previously, the different playback times may be based on adesired acoustic characteristic in the environment. For example, thedesired acoustic characteristic may include a type of playback, such as:monophonic, stereophonic and/or multichannel sound. Monophonic sound mayinclude one or more audio signals that contain no amplitude (or level)and arrival time/phase information that replicates or simulatesdirectional cues.

Moreover, stereophonic sound may include two independent audio-signalchannels, and the audio signals may have a specific amplitude and phaserelationship with each other so that, during the playback operation,there is an apparent image of the original sound source. In general, theaudio signals for both channels may provide coverage over most or all ofthe environment. By adjusting the relative amplitudes and/or phases ofthe audio channels, the sweet spot may be moved to follow the determinedlocation of at least the listener. However, the amplitude differencesand arrival time differences (the directional cues) may need to be smallenough that the stereo image and localization are both maintained.Otherwise, the image may collapse and only one or the other audiochannel is heard.

Note that the audio channels in stereophonic sound may need to have thecorrect absolute phase response. This means that an audio signal with apositive pressure waveform at the input to the system may need to havethe same positive pressure waveform at the output from one of speakers118. Therefore, a drum, which, when struck, produces a positive pressurewaveform at a microphone may need to produce a positive pressurewaveform in the environment. Alternatively, if the absolute polarity isflipped the wrong way, the audio image may not be stable. In particular,the listener may not find or perceive a stable audio image. Instead, theaudio image may wander and may localize at speakers 118.

Furthermore, multichannel sound may include left, center and right audiochannels. For example, these channels may allow monophonic speechreinforcement and music or sound effect cues to be localized or mixedwith a particular perspective, with stereo or stereo-like imaging. Thus,the three audio channels may provide coverage over most or all of theentire environment while maintaining amplitude and directional cues, aswas the case for monophonic or stereophonic sound.

Alternatively or additionally, the desired acoustic characteristic mayinclude an acoustic radiation pattern. The desired acoustic radiationpattern may be a function of the reverberation time in the environment.For example, the reverberation time may change depending on the numberof people in the environment, the type and amount of furniture in theenvironment, whether or not the curtains are open or closed, whether ornot a window is open or closed, etc. When the reverberation time islonger or is increased, the desired acoustic radiation pattern may bemore directed, so that the sound is steered or beamed to a listener(thereby reducing the reverberation). In some embodiments, the desiredacoustic characteristic includes intelligibility of words.

While the preceding discussion illustrated techniques that can be usedto dynamically track location 1610 of listener 1612 (or portableelectronic device 110), these techniques may be used to determine thelocation of an electronic device (such as a speaker 118-1) in theenvironment.

Another approach for improving the acoustic experience is to dynamicallyaggregate electronic devices into groups and/or to adapt thecoordination based on the groups. This is shown in FIG. 17, whichpresents a flow diagram illustrating a method 1700 for aggregatingelectronic devices. Note that method 1700 may be performed by an A/Vhub, such as A/V hub 112 (FIG. 1). During operation, the A/V hub (suchas a control circuit or control logic, e.g., a processor executing aprogram module, in the A/V hub) may measure sound (operation 1710)output by electronic devices (such as speakers 118) in an environmentusing one or more acoustic transducers, where the sound corresponds toaudio content. For example, the measured sound may include the soundpressure.

Then, the A/V hub may aggregate the electronic devices (operation 1712)into two or more subsets based on the measured sound. Note that thedifferent subsets may be located in different rooms in the environment.Moreover, at least one of the subsets may playback different audiocontent than a remainder of the subsets. Furthermore, the aggregation ofthe electronic devices into the two or more subsets may be based on: thedifferent audio content; an acoustic delay of the measured sound; and/ora desired acoustic characteristic in the environment. In someembodiments, electronic devices in the subsets and/or geographiclocations or regions associated with the subsets are not predefined.Instead, the A/V hub may dynamically aggregate the subsets.

Moreover, the A/V hub may determine playback timing information(operation 1714) for the subsets, where the playback timing informationspecifies playback times when the electronic devices in a given subsetare to playback the audio content.

Next, the A/V hub may transmit, using wireless communication, one ormore frames (operation 1716) or packets that include the audio contentand playback timing information to the electronic devices, where theplayback times of the electronic devices in at least the given subsethave a temporal relationship so that the playback of the audio contentby the electronic devices in the given subset is coordinated.

In some embodiments, the A/V hub optionally performs one or moreadditional operations (operation 1718). For example, the A/V hub maycalculate an estimated location of at least a listener relative to theelectronic devices, and the aggregation of the electronic devices intothe two or more subsets may be based on the estimated location of atleast the listener. This may help ensure that the listener has animproved acoustic experience, with reduced acoustic cross-talk from theother subset(s).

Moreover, the A/V hub may modify the measured sound based on apredetermined (or dynamically determined) acoustic transfer function ofthe environment in at least a band of frequencies (such as 100-20,000Hz, which is a nonlimiting example). This may allow the A/V hub todetermine the original output sound without the spectral filtering ordistortions associated with the environment, which may allow the A/V hubto make better decisions when aggregating the subsets.

Furthermore, the A/V hub may determine playback volumes for the subsetsthat are used when the subsets playback the audio content, and the oneor more frames or packets may include information that specifies theplayback volumes. For example, a playback volume for at least one of thesubsets may be different than the playback volumes of a remainder of thesubsets. Alternatively or additionally, the playback volumes may reduceacoustic cross-talk among the two or more subsets so that listeners aremore likely to hear the sound output by the subset to which they areproximate or closest.

FIG. 18 is a drawing illustrating communication among portableelectronic device 110, A/V hub 112, and speakers 118. In particular,processor 1810 may instruct 1812 one or more acoustic transducers 1814in A/V hub 112 to perform measurements 1816 of sound associated withspeakers 118. Then, based on measurements 1816, processor 1810 mayaggregate speakers 118 into two or more subsets 1818.

Moreover, processor 1810 may determine playback timing information 1820for subsets 1818, wherein the playback timing information 1820 specifiesplayback times when speakers 118 in a given subset are to playback audiocontent 1822. Note that processor 1810 may access audio content 1822 inmemory 1824.

Next, processor 1810 may instruct interface circuit 1826 to transmitframes or packets 1828 to speakers 118 with playback timing information1820 and audio content 1822. (However, in some embodiments, playbacktiming information 1820 and audio content 1822 are transmitted usingseparate or different frames or packets.)

After receiving the one or more frames or packets 1826, an interfacecircuit in speaker 118-3 may provide playback timing information 1820and audio content 1822 to a processor. This processor may executesoftware that performs a playback operation 1830. For example, theprocessor may store audio content 1822 in a queue in memory. In theseembodiments, playback operation 1830 includes outputting audio content1822 from the queue, including driving one or more of acoustictransducers based on audio content 1822 so speaker 118-3 outputs soundat a time specified by the playback timing information 1820. Note thatthe playback times of speakers 118 in at least the given subset have atemporal relationship so that the playback of audio content 1822 by thespeakers 118 in the given subset is coordinated.

In an exemplary embodiment, the communication technique is used toaggregate speakers 118 into subsets. FIG. 19 presents a drawingillustrating aggregating speakers 118, which may be in the same ordifferent rooms in an environment. A/V hub 112 may measure sound 1910output by speakers 118. Based on these measurements, A/V hub 112 mayaggregate speakers 118 into subsets 1912. For example, the subsets 1912may be aggregated based on sound intensity and/or acoustic delay, sothat proximate speakers are aggregated together. In particular, speakersthat have the highest acoustic intensity or similar acoustic delay maybe aggregated together. In order to facilitate the aggregation, speakers118 may wirelessly transmit and/or acoustically output identificationinformation or acoustic-characterization patterns outside of the rangeof human hearing. For example, the acoustic-characterization patternsmay include pulses. However, a variety of temporal, frequency and/ormodulation techniques may be used, including: amplitude modulation,frequency modulation, phase modulation, etc. Alternatively oradditionally, A/V hub 112 may instruct each of speakers 118 to, one at atime, dither the playback times or phase of their output sound, so thatA/V hub 112 can associate the measured sound with particular speakers.Moreover, the measured sound 1910 may be corrected using an acoustictransfer function of an environment, so that the impact of reflectionsand filtering (or distortion) is removed prior to aggregating speakers118. In some embodiments, the speakers 118 are aggregated based, atleast in part, on locations 1914 of speakers 118, which may bedetermined using one or more of the aforementioned techniques (such asusing wireless ranging). In this way, subsets 1912 may be dynamicallymodified as one or more listeners repositions speakers 118 in theenvironment.

Then, A/V hub 112 may transmit one or more frames or packets (such aspacket 1916) that include additional audio content 1918 (such as music)and playback timing information 1920 to speakers 118 in at least one ofsubsets 1912 (such as subset 1912-1), where the playback timinginformation 1920 may specify playback times when speakers 118 in subset1912-1 are to playback the additional audio content 1918. Thus, thecommunication technique may be used to dynamically select subsets 1912,e.g., based on a location of a listener and/or a desired acousticcharacteristic in an environment that includes A/V hub 112 and speakers118.

Another approach for improving the acoustic experience is to dynamicallyequalize audio based on acoustic monitoring in an environment. FIG. 20presents a flow diagram illustrating a method 2000 for determiningequalized audio content, which may be performed by an A/V hub, such asA/V hub 112 (FIG. 1). During operation, the A/V hub (such as a controlcircuit or control logic, e.g., a processor executing a program module,in the A/V hub) may measure sound (operation 2010) output by electronicdevices (such as speakers 118) in the environment using one or moreacoustic transducers, where the sound corresponds to audio content. Forexample, the measured sound may include the sound pressure.

Then, the A/V hub may compare the measured sound to a desired acousticcharacteristic (operation 2012) at a first location in the environmentbased on the first location, a second location of the A/V hub, and apredetermined or dynamically determined acoustic transfer function ofthe environment in at least a band of frequencies (such as 100-20,000kHz, which is a nonlimiting example). Note that the comparison may beperformed in the time domain and/or in the frequency domain. In order toperform the comparison, the A/V hub may calculate the acousticcharacteristic (such as an acoustic transfer function or a modalresponse) at the first location and/or the second location, and maycorrect the measured sound for filtering or distortions in theenvironment using the calculated acoustic characteristic. Using theacoustic transfer function as an example, this calculation may involvethe use of a Green's function technique to compute the acoustic responseof the environment as a function of location with one or more point ordistributed acoustic sources at predefined or known location(s) in theenvironment. Note that the acoustic transfer function at the firstlocation and the correction may depend on the integrated acousticbehavior of the environment (and, thus, the second location and/orlocations of acoustic sources, such as speakers 118, in theenvironment). Therefore, the acoustic transfer function may includeinformation specifying the location(s) in the environment where theacoustic transfer function was determined (e.g., the second location)and/or the location(s) of an acoustic source in the environment (such asthe location of at least one of the electronic devices).

Moreover, the A/V hub may determine equalized audio content (operation2014) based on the comparison and the audio content. Note that thedesired acoustic characteristic may be based on a type of audioplayback, such as: monophonic, stereophonic and/or multichannel.Alternatively or additionally, the desired acoustic characteristic mayinclude an acoustic radiation pattern. The desired acoustic radiationpattern may be a function of the reverberation time in the environment.For example, the reverberation time may change depending on the numberof people in the environment, the type and amount of furniture in theenvironment, whether or not the curtains are open or closed, whether ornot a window is open or closed, etc. When the reverberation time islonger or is increased, the desired acoustic radiation pattern may bemore directed, so that the sound associated with the equalized audiocontent is steered or beamed to a listener (thereby reducing thereverberation). Consequently, in some embodiments the equalization is acomplex function that modifies the amplitude and/or phase in the audiocontent. Moreover, the desired acoustic characteristic may includereducing room resonances or room modes by reducing the energy in theassociated low frequencies in the acoustic content. Note that in someembodiments, the desired acoustic characteristic includesintelligibility of words. Thus, the target (the desired acousticcharacteristic) may be used to adapt the equalization of the audiocontent.

Next, the A/V hub may transmit, using wireless communication, one ormore frames (operation 2016) or packets that include the equalized audiocontent to the electronic devices to facilitate output by the electronicdevices of additional sound, which corresponds to the equalized audiocontent.

In some embodiments, the A/V hub optionally performs one or moreadditional operations (operation 2018). For example, the first locationmay include an estimated location of a listener relative to theelectronic devices, and the A/V hub may calculate the estimated locationof the listener. In particular, the estimated location of the listenermay use one or more of the aforementioned techniques for dynamicallydetermining the location of the listener. Thus, the A/V hub maycalculate the estimated location of the listener based on the soundmeasurements. Alternatively or additionally, the A/V hub may:communicate with another electronic device; and may calculate theestimated location of the listener based on the communication with theother electronic device. In some embodiments, the communication with theother electronic device includes wireless ranging, and the estimatedlocation may be calculated based on the wireless ranging and an angle ofarrival of wireless signals from the other electronic device.Furthermore, the A/V hub may perform time-of-flight measurements, andmay calculate the estimated location of the listener based on thetime-of-flight measurements. In some embodiments, the dynamicequalization allows the ‘sweet spot’ in the environment to be adaptedbased on the location of the listener. Note that the A/V hub maydetermine the number of listeners in the environment and/or thelocations of the listeners, and the dynamic equalization may adapt thesound so that the listeners (or a majority of the listeners) have thedesired acoustic characteristic when listening to the equalized audiocontent.

Moreover, the A/V hub may communicate with other electronic devices inthe environment and may receive (separately from or in conjunction withthe sound measurements) additional sound measurements of the environmentfrom the other electronic devices. Then, the A/V hub may perform one ormore additional comparisons of the additional sound measurements to thedesired acoustic characteristic at the first location in the environmentbased on one or more third locations of the other electronic devices(such as the locations of speakers 118) and the predetermined ordynamically determined acoustic transfer function of the environment inat least the band of frequencies, and the equalized audio content isfurther determined based on the one or more additional comparisons. Insome embodiments, the A/V hub determines the one or more third locationsbased on the communication with the other electronic devices. Forexample, the communication with the other electronic devices may includewireless ranging, and the one or more third locations may be calculatedbased on the wireless ranging and angles of arrival of wireless signalsfrom the other electronic devices. Alternatively or additionally, theA/V hub may receive information specifying the third locations from theother electronic devices. Thus, the locations of the other electronicdevices may be determined using one or more of the aforementionedtechniques for determining the location of an electronic device in theenvironment.

Furthermore, the A/V hub may determine playback timing information thatspecifies playback times when the electronic devices playback theequalized audio content, and the one or more frames or packets mayinclude the playback timing information. In these embodiments, theplayback times of the electronic devices have a temporal relationship sothat the playback of the audio content by the electronic devices iscoordinated.

FIG. 21 is a drawing illustrating communication among portableelectronic device 110, A/V hub 112, and speakers 118. In particular,processor 2110 may instruct 2112 one or more acoustic transducers 2114in A/V hub 112 to measure sound 2116 associated with speakers 118 andcorresponding to audio content 2118. Then, processor 21110 may compare2120 the measured sound 2116 to a desired acoustic characteristic 2122at a first location in the environment based on the first location, asecond location of A/V hub 112, and a predetermined or dynamicallydetermined acoustic transfer 2124 function of the environment in atleast a band of frequencies (which may be accessed in memory 2128).

Moreover, processor 2110 may determine equalized audio content 2126based on comparison 2120 and audio content 2118, which may be accessedin memory 2128. Note that processor 2110 may know, in advance, audiocontent 2118 being output by speakers 118.

Next, processor 2110 may determine playback timing information 2130,wherein the playback timing information 2130 specifies playback timeswhen speakers 118 are to playback equalized audio content 2126.

Furthermore, processor 2110 may instruct interface circuit 2132 totransmit one or more frames or packets 2134 to speakers 118 withplayback timing information 2130 and equalized audio content 2126.(However, in some embodiments, playback timing information 2130 andaudio content 2126 are transmitted using separate or different frames orpackets.)

After receiving the one or more frames or packets 2134, an interfacecircuit in one of speakers 118 (such as speaker 118-1) may provideplayback timing information 2130 and equalized audio content 2126 to aprocessor. This processor may execute software that performs a playbackoperation. For example, the processor may store equalized audio content2126 in a queue in memory. In these embodiments, the playback operationincludes outputting equalized audio content 2126 from the queue,including driving one or more of acoustic transducers based on equalizedaudio content 2126 so speaker 118-1 outputs sound at a time specified bythe playback timing information 2130. Note that the playback times ofspeakers 118 have a temporal relationship so that the playback ofequalized audio content 2126 by the speakers 118 is coordinated.

In an exemplary embodiment, the communication technique is used todynamically equalize audio content. FIG. 22 presents a drawingillustrating determining equalized audio content using speakers 118. Inparticular, A/V hub 112 may measure sound 2210, corresponding to audiocontent, which is output by speakers 118. Alternatively or additionally,portable electronic device 110 and/or at least some of speakers 118 maymeasure sound 2210 and may provide information specifying themeasurements to A/V hub 112 in frames or packets 2212.

Then, A/V hub 112 may compare the measured sound 2210 to a desiredacoustic characteristic at a location 2214 in the environment (such as adynamic location of one or more listeners, which may also be thelocation of portable electronic device 110) based on location 2214,location 2216 of A/V hub 112, locations 2218 of speakers 118, and/or apredetermined or dynamically determined acoustic transfer function (or,more generally, an acoustic characteristic) of the environment in atleast a band of frequencies. For example, A/V hub 112 may calculate theacoustic transfer function at location 2214, 2216 and/or 2218. As notedpreviously, this calculation may involve the use of a Green's functiontechnique to compute the acoustic response at locations 2214, 2216and/or 2218. Alternatively or additionally, the calculation may involveinterpolation (such as minimum bandwidth interpolation) of apredetermined acoustic transfer function at different locations in theenvironment that locations 2214, 2216 and/or 2218. Then, A/V hub 112 maycorrect the measured sound 2210 based on the computed and/orinterpolated acoustic transfer function (and, more generally, theacoustic characteristic).

In this way, the communication technique may be used to compensate forsparse sampling when the acoustic transfer function was originallydetermined.

Moreover, A/V hub 112 may determine equalized audio content based on thecomparison and the audio content. For example, A/V hub 112 may modifythe spectral content and/or phase of the audio content as a function offrequency in a range of frequencies (such as 100-10,000 or 20,000 Hz) toachieve the desired acoustic characteristic.

Next, A/V hub 112 may transmit one or more frames or packets thatinclude the equalized audio content (such as music) and playback timinginformation to speakers 118 (such as packet 2220 with equalized audiocontent 2222 and playback timing information 2224), where the playbacktiming information may specify playback times when speakers 118 are toplayback the equalized audio content.

In this way, the communication technique may allow the sound output byspeakers 118 to adapt to changes in location 2214 of one or morelisteners (such as an average or mean location, a location correspondingto a majority of the listeners, an average location of a largest subsetof the listeners for which the desired acoustic characteristic can beachieved given the audio content and the acoustic transfer function orthe acoustic characteristics of the environment, etc.). This may allowthe sweet spot in stereophonic sound to track motion of the one or morelisteners and/or changes in the number of listeners in the environment(which may be determined by A/V hub 112 using one or more of theaforementioned techniques). Alternatively or additionally, thecommunication technique may allow the sound output by speakers 118 toadapt to changes in the audio content and/or in the desired acousticcharacteristic. For example, depending on the type of audio content(such as a type of music), the one or more listeners may want or desirea big or broad sound (with diverging sound waves corresponding to anapparently physically extended acoustic source) or an apparently narrowor point source. Thus, the communication technique may allow the audiocontent to be equalized according to a desired psychoacoustic experienceof the one or more listeners. Note that the desired acousticcharacteristic or the desired psychoacoustic experience may beexplicitly specified by one or more of the listeners (such by using auser interface on portable electronic device 110) or may be determinedor inferred indirectly without user action (such as based on the type ofmusic or prior acoustic preferences of the one or more listeners thatare stored in a listening history).

In some embodiments of methods 200 (FIG. 2), 500 (FIG. 5), 800 (FIG. 8),1100 (FIG. 11), 1400 (FIG. 14), 1700 (FIG. 17) and/or 2000 (FIG. 20)there are additional or fewer operations. Moreover, the order of theoperations may be changed, and/or two or more operations may be combinedinto a single operation. Furthermore, one or more operations may bemodified.

We now describe embodiments of an electronic device. FIG. 23 presents ablock diagram illustrating an electronic device 2300, such as portableelectronic device 110, A/V hub 112, one of A/V display devices 114,receiver device 116 or one of speakers 118 in FIG. 1. This electronicdevice includes processing subsystem 2310, memory subsystem 2312,networking subsystem 2314, optional feedback subsystem 2334, andoptional monitoring subsystem 2336. Processing subsystem 2310 includesone or more devices configured to perform computational operations. Forexample, processing subsystem 2310 can include one or moremicroprocessors, application-specific integrated circuits (ASICs),microcontrollers, programmable-logic devices, and/or one or more digitalsignal processors (DSPs). One or more of these components in processingsubsystem are sometimes referred to as a ‘control circuit.’ In someembodiments, processing subsystem 2310 includes a ‘control mechanism’ ora ‘means for processing’ that perform at least some of the operations inthe communication technique.

Memory subsystem 2312 includes one or more devices for storing dataand/or instructions for processing subsystem 2310 and networkingsubsystem 2314. For example, memory subsystem 2312 can include dynamicrandom access memory (DRAM), static random access memory (SRAM), and/orother types of memory. In some embodiments, instructions for processingsubsystem 2310 in memory subsystem 2312 include: one or more programmodules or sets of instructions (such as program module 2322 oroperating system 2324), which may be executed by processing subsystem2310. Note that the one or more computer programs or program modules mayconstitute a computer-program mechanism. Moreover, instructions in thevarious modules in memory subsystem 2312 may be implemented in: ahigh-level procedural language, an object-oriented programming language,and/or in an assembly or machine language. Furthermore, the programminglanguage may be compiled or interpreted, e.g., configurable orconfigured (which may be used interchangeably in this discussion), to beexecuted by processing subsystem 2310.

In addition, memory subsystem 2312 can include mechanisms forcontrolling access to the memory. In some embodiments, memory subsystem2312 includes a memory hierarchy that comprises one or more cachescoupled to a memory in electronic device 2300. In some of theseembodiments, one or more of the caches is located in processingsubsystem 2310.

In some embodiments, memory subsystem 2312 is coupled to one or morehigh-capacity mass-storage devices (not shown). For example, memorysubsystem 2312 can be coupled to a magnetic or optical drive, asolid-state drive, or another type of mass-storage device. In theseembodiments, memory subsystem 2312 can be used by electronic device 2300as fast-access storage for often-used data, while the mass-storagedevice is used to store less frequently used data.

Networking subsystem 2314 includes one or more devices configured tocouple to and communicate on a wired and/or wireless network (i.e., toperform network operations), including: control logic 2316, interfacecircuits 2318 and associated antennas 2320. (While FIG. 23 includesantennas 2320, in some embodiments electronic device 2300 includes oneor more nodes, such as nodes 2308, e.g., pads, which can be coupled toantennas 2320. Thus, electronic device 2300 may or may not includeantennas 2320.) For example, networking subsystem 2314 can include aBluetooth networking system, a cellular networking system (e.g., a 3G/4Gnetwork such as UMTS, LTE, etc.), a universal serial bus (USB)networking system, a networking system based on the standards describedin IEEE 802.11 (e.g., a Wi-Fi networking system), an Ethernet networkingsystem, and/or another networking system. Note that the combination of agiven one of interface circuits 2318 and at least one of antennas 2320may constitute a radio. In some embodiments, networking subsystem 2314includes a wired interface, such as HDMI interface 2330.

Networking subsystem 2314 includes processors, controllers,radios/antennas, sockets/plugs, and/or other devices used for couplingto, communicating on, and handling data and events for each supportednetworking system. Note that mechanisms used for coupling to,communicating on, and handling data and events on the network for eachnetwork system are sometimes collectively referred to as a ‘networkinterface’ for the network system. Moreover, in some embodiments a‘network’ between the electronic devices does not yet exist. Therefore,electronic device 2300 may use the mechanisms in networking subsystem2314 for performing simple wireless communication between the electronicdevices, e.g., transmitting advertising or beacon frames or packetsand/or scanning for advertising frames or packets transmitted by otherelectronic devices as described previously.

Within electronic device 2300, processing subsystem 2310, memorysubsystem 2312, networking subsystem 2314, optional feedback subsystem2334 and optional monitoring subsystem 2336 are coupled together usingbus 2328. Bus 2328 may include an electrical, optical, and/orelectro-optical connection that the subsystems can use to communicatecommands and data among one another. Although only one bus 2328 is shownfor clarity, different embodiments can include a different number orconfiguration of electrical, optical, and/or electro-optical connectionsamong the subsystems.

In some embodiments, electronic device 2300 includes a display subsystem2326 for displaying information on a display (such as a request toclarify an identified environment), which may include a display driver,an I/O controller and the display. Note that a wide variety of displaytypes may be used in display subsystem 2326, including: atwo-dimensional display, a three-dimensional display (such as aholographic display or a volumetric display), a head-mounted display, aretinal-image projector, a heads-up display, a cathode ray tube, aliquid-crystal display, a projection display, an electroluminescentdisplay, a display based on electronic paper, a thin-film transistordisplay, a high-performance addressing display, an organiclight-emitting diode display, a surface-conduction electronic-emitterdisplay, a laser display, a carbon-nanotube display, a quantum-dotdisplay, an interferometric modulator display, a multi-touch touchscreen(which is sometimes referred to as a touch-sensitive display), and/or adisplay based on another type of display technology or physicalphenomenon.

Furthermore, optional feedback subsystem 2334 may include one or moresensor-feedback mechanisms or devices, such as: a vibration mechanism ora vibration actuator (e.g., an eccentric-rotating-mass actuator or alinear-resonant actuator), a light, one or more speakers, etc., whichcan be used to provide feedback to a user of electronic device 2300(such as sensory feedback). Alternatively or additionally, optionalfeedback subsystem 2334 may be used to provide a sensory input to theuser. For example, the one or more speakers may output sound, such asaudio. Note that the one or more speakers may include an array oftransducers that can be modified to adjust a characteristic of the soundoutput by the one or more speakers, such as a phased-array of acoustictransducers. This capability may allow the one or more speakers tomodify the sound in an environment to achieve a desired acousticexperience for a user, such as by changing equalization or spectralcontent, phase and/or a direction of the propagating sound waves.

In some embodiments, optional monitoring subsystem 2336 includes one ormore acoustic transducers 2338 (such as one or more microphones, aphased-array, etc.) that monitor sound in the environment that includeselectronic device 2300. The acoustic monitoring may allow electronicdevice 2300 to acoustically characterize the environment, acousticallycharacterize sound output by speakers in the environment (such as soundcorresponding to audio content), determine a location of a listener,determine a location of a speaker in the environment and/or measuresound from one or more speakers that correspond to one or moreacoustic-characterization patterns (which may be used to coordinateplayback of audio content). Additionally, optional monitoring subsystem2336 may include location transducers 2340 that can be used to determinea location of a listener or an electronic device (such as a speaker) inthe environment.

Electronic device 2300 can be (or can be included in) any electronicdevice with at least one network interface. For example, electronicdevice 2300 can be (or can be included in): a desktop computer, a laptopcomputer, a subnotebook/netbook, a server, a tablet computer, asmartphone, a cellular telephone, a smartwatch, a consumer-electronicdevice (such as a television, a set-top box, audio equipment, a speaker,video equipment, etc.), a remote control, a portable computing device,an access point, a router, a switch, communication equipment, testequipment, and/or another electronic device.

Although specific components are used to describe electronic device2300, in alternative embodiments, different components and/or subsystemsmay be present in electronic device 2300. For example, electronic device2300 may include one or more additional processing subsystems, memorysubsystems, networking subsystems, and/or display subsystems. Moreover,while one of antennas 2320 is shown coupled to a given one of interfacecircuits 2318, there may be multiple antennas coupled to the given oneof interface circuits 2318. For example, an instance of a 3×3 radio mayinclude three antennas. Additionally, one or more of the subsystems maynot be present in electronic device 2300. Furthermore, in someembodiments, electronic device 2300 may include one or more additionalsubsystems that are not shown in FIG. 23. Also, although separatesubsystems are shown in FIG. 23, in some embodiments, some or all of agiven subsystem or component can be integrated into one or more of theother subsystems or component(s) in electronic device 2300. For example,in some embodiments program module 2322 is included in operating system2324.

Moreover, the circuits and components in electronic device 2300 may beimplemented using any combination of analog and/or digital circuitry,including: bipolar, PMOS and/or NMOS gates or transistors. Furthermore,signals in these embodiments may include digital signals that haveapproximately discrete values and/or analog signals that have continuousvalues. Additionally, components and circuits may be single-ended ordifferential, and power supplies may be unipolar or bipolar.

An integrated circuit may implement some or all of the functionality ofnetworking subsystem 2314, such as one or more radios. Moreover, theintegrated circuit may include hardware and/or software mechanisms thatare used for transmitting wireless signals from electronic device 2300and receiving signals at electronic device 2300 from other electronicdevices. Aside from the mechanisms herein described, radios aregenerally known in the art and hence are not described in detail. Ingeneral, networking subsystem 2314 and/or the integrated circuit caninclude any number of radios.

In some embodiments, networking subsystem 2314 and/or the integratedcircuit include a configuration mechanism (such as one or more hardwareand/or software mechanisms) that configures the radios to transmitand/or receive on a given channel (e.g., a given carrier frequency). Forexample, in some embodiments, the configuration mechanism can be used toswitch the radio from monitoring and/or transmitting on a given channelto monitoring and/or transmitting on a different channel. (Note that‘monitoring’ as used herein comprises receiving signals from otherelectronic devices and possibly performing one or more processingoperations on the received signals, e.g., determining if the receivedsignal comprises an advertising frame or packet, calculating aperformance metric, performing spectral analysis, etc.) Furthermore,networking subsystem 2314 may include at least one port (such as an HDMIport 2332) to receive and/or provide the information in the data streamto at least one of A/V display devices 114 (FIG. 1), at least one ofspeakers 118 (FIG. 1) and/or at least one of content sources 120 (FIG.1).

While a communication protocol compatible with Wi-Fi was used as anillustrative example, the described embodiments may be used in a varietyof network interfaces. Furthermore, while some of the operations in thepreceding embodiments were implemented in hardware or software, ingeneral the operations in the preceding embodiments can be implementedin a wide variety of configurations and architectures. Therefore, someor all of the operations in the preceding embodiments may be performedin hardware, in software or both. For example, at least some of theoperations in the communication technique may be implemented usingprogram module 2322, operating system 2324 (such as drivers forinterface circuits 2318) and/or in firmware in interface circuits 2318.Alternatively or additionally, at least some of the operations in thecommunication technique may be implemented in a physical layer, such ashardware in interface circuits 2318.

Moreover, while the preceding embodiments included a touch-sensitivedisplay in the portable electronic device that the user touches (e.g.,with a finger or digit, or a stylus), in other embodiments the userinterface is display on a display in the portable electronic device andthe user interacts with the user interface without making contact ortouching the surface of the display. For example, the user's interact(s)with the user interface may be determined using time-of-flightmeasurements, motion sensing (such as a Doppler measurement) or anothernon-contact measurement that allows the position, direction of motionand/or speed of the user's finger or digit (or a stylus) relative toposition(s) of one or more virtual command icons to be determined. Inthese embodiments, note that the user may activate a given virtualcommand icon by performing a gesture (such as ‘tapping’ their finger inthe air without making contact with the surface of the display). In someembodiments, the user navigates through the user interface and/oractivates/deactivates functions of one of the components in system 100(FIG. 1) using spoken commands or instructions (i.e., via voicerecognition) and/or based on where they are looking at one a display inportable electronic device 110 or on one of A/V display devices 114 inFIG. 1 (e.g., by tracking the user's gaze or where the user is looking).

Furthermore, while A/V hub 112 (FIG. 1) were illustrated as separatecomponents from A/V display devices 114 (FIG. 1), in some embodiments anA/V hub and an A/V display device are combined into a single componentor a single electronic device.

While the preceding embodiments illustrated the communication techniquewith audio and/or video content (such as HDMI content), in otherembodiments the communication technique is used in the context of anarbitrary type of data or information. For example, the communicationtechnique may be used with home-automation data. In these embodiments,A/V hub 112 (FIG. 1) may facilitate communication among and control of awide variety of electronic devices. Thus, A/V hub 112 (FIG. 1) and thecommunication technique may be used to facilitate or implement servicesin the so-called Internet of things.

In the preceding description, we refer to ‘some embodiments.’ Note that‘some embodiments’ describes a subset of all of the possibleembodiments, but does not always specify the same subset of embodiments.

The foregoing description is intended to enable any person skilled inthe art to make and use the disclosure, and is provided in the contextof a particular application and its requirements. Moreover, theforegoing descriptions of embodiments of the present disclosure havebeen presented for purposes of illustration and description only. Theyare not intended to be exhaustive or to limit the present disclosure tothe forms disclosed. Accordingly, many modifications and variations willbe apparent to practitioners skilled in the art, and the generalprinciples defined herein may be applied to other embodiments andapplications without departing from the spirit and scope of the presentdisclosure. Additionally, the discussion of the preceding embodiments isnot intended to limit the present disclosure. Thus, the presentdisclosure is not intended to be limited to the embodiments shown, butis to be accorded the widest scope consistent with the principles andfeatures disclosed herein.

What is claimed is:
 1. A coordination device, comprising: one or moreacoustic transducers configured to measure sound in an environment; oneor more nodes configured to communicatively couple to one or moreantennas; an interface circuit communicatively coupled to the one ormore nodes, wherein the coordination device is configured to: measurethe sound using the one or more acoustic transducers, wherein the soundcorresponds to one or more acoustic-characterization patterns; calculatecurrent time offsets between clocks in electronic devices and a clock inthe coordination device based on the measured sound, one or more timeswhen the electronic devices output the sound and the one or moreacoustic-characterization patterns; and transmit, to the one or morenodes, one or more frames that include audio content and playback timinginformation for the electronic devices, wherein the playback timinginformation specifies playback times when the electronic devices are toplayback the audio content based on the current time offsets, andwherein the playback times of the electronic devices have a temporalrelationship so that the playback of the audio content by the electronicdevices is coordinated.
 2. The coordination device of claim 1, whereinthe measured sound includes information that specifies the one or moretimes when the electronic devices output the sound; and wherein the oneor more times correspond to the clocks in the electronic devices.
 3. Thecoordination device of claim 1, wherein the coordination device isfurther configured to provide, to the one or more nodes, one or moretimes when the electronic devices are to output the sound; and whereinthe one or more times correspond to the clock in the coordinationdevice.
 4. The coordination device of claim 1, wherein a givenelectronic device outputs the sound at a different time in the one ormore times than those used by a remainder of the electronic devices. 5.The coordination device of claim 1, wherein the sound output by a givenelectronic device corresponds to a given acoustic-characterizationpattern, which is different from those used by a remainder of theelectronic devices.
 6. The coordination device of claim 1, wherein theacoustic-characterization patterns include pulses.
 7. The coordinationdevice of claim 1, wherein the sound is in a range of frequenciesoutside of human hearing.
 8. The coordination device of claim 1, whereinthe coordination device is further configured to modify the measuredsound based on an acoustic transfer function of the environment in atleast a band of frequencies.
 9. The coordination device of claim 1,wherein the temporal relationship has a non-zero value, so that at leastsome of the electronic devices are instructed to playback the audiocontent with a phase relative to each other by using different values ofthe playback times.
 10. The coordination device of claim 9, wherein thedifferent playback times are based on acoustic characterization of theenvironment.
 11. The coordination device of claim 9, wherein thedifferent playback times are based on a desired acoustic characteristicin the environment.
 12. The coordination device of claim 9, wherein thedifferent playback times are based on an estimated location of alistener or another electronic device associated with the listenerrelative to the electronic devices.
 13. A non-transitorycomputer-readable storage medium for use with a coordination device, thecomputer-readable storage medium storing a program module that, whenexecuted by the coordination device, causes the coordination device tocoordinate playback of audio content by carrying out one or moreoperations that comprise: measuring sound, in an environment, using oneor more acoustic transducers, wherein the sound corresponds to one ormore acoustic-characterization patterns; calculating current timeoffsets between clocks in electronic devices and a clock in thecoordination device based on the measured sound, one or more times whenthe electronic devices output the sound and the one or moreacoustic-characterization patterns; and transmitting, to one or morenodes in the coordination device that are communicatively coupled to oneor more antennas, one or more frames that include the audio content andplayback timing information for the electronic devices, wherein theplayback timing information specifies playback times when the electronicdevices are to playback the audio content based on the current timeoffsets, and wherein the playback times of the electronic devices have atemporal relationship so that the playback of the audio content by theelectronic devices is coordinated.
 14. The computer-readable storagemedium of claim 13, wherein the measured sound includes information thatspecifies the one or more times when the electronic devices output thesound; and wherein the one or more times correspond to the clocks in theelectronic devices.
 15. The computer-readable storage medium of claim13, wherein the one or more operation comprise providing, to the one ormore nodes, one or more times when the electronic devices are to outputthe sound; and wherein the one or more times correspond to the clock inthe coordination device.
 16. The computer-readable storage medium ofclaim 13, wherein a given electronic device outputs the sound at adifferent time in the one or more times than those used by a remainderof the electronic devices.
 17. The computer-readable storage medium ofclaim 13, wherein the sound output by a given electronic devicecorresponds to a given acoustic-characterization pattern, which isdifferent from those used by a remainder of the electronic devices. 18.The computer-readable storage medium of claim 13, wherein the sound isin a range of frequencies outside of human hearing.
 19. Thecomputer-readable storage medium of claim 13, wherein the one or moreoperations comprise modifying the measured sound based on an acoustictransfer function of the environment in at least a band of frequencies.20. A method for coordinating playback of audio content, comprising: bya coordination device: measuring sound, in an environment, using one ormore acoustic transducers, wherein the sound corresponds to one or moreacoustic-characterization patterns; calculating current time offsetsbetween clocks in electronic devices and a clock in the coordinationdevice based on the measured sound, one or more times when theelectronic devices output the sound and the one or moreacoustic-characterization patterns; and transmitting, to one or morenodes in the coordination device that are communicatively coupled to oneor more antennas, one or more frames that include audio content andplayback timing information to the electronic devices, wherein theplayback timing information specifies playback times when the electronicdevices are to playback the audio content based on the current timeoffsets, and wherein the playback times of the electronic devices have atemporal relationship so that the playback of the audio content by theelectronic devices is coordinated.